getting students back on track: persistent effects of

doi.org/10.26434/chemrxiv.9775910.v4

Getting Students Back on Track: Persistent Effects of FlippingAccelerated Organic Chemistry on Student Achievement, StudyStrategies, and Perceptions of InstructionLynn C. Reimer, Kameryn Denaro, Wenliang He, Renee Link

Submitted date: 05/02/2021 • Posted date: 08/02/2021Licence: CC BY-NC-ND 4.0Citation information: Reimer, Lynn C.; Denaro, Kameryn; He, Wenliang; Link, Renee (2019): Getting StudentsBack on Track: Persistent Effects of Flipping Accelerated Organic Chemistry on Student Achievement, StudyStrategies, and Perceptions of Instruction. ChemRxiv. Preprint. https://doi.org/10.26434/chemrxiv.9775910.v4

Converting a first-term, accelerated summer organic chemistry course to a flipped format reduced theachievement gap in the flipped course and in the second-term traditional lecture course betweenNon-Repeaters taking an accelerated course to “get ahead” and Repeaters taking the course to “get back ontrack.” The difference in final exam performance in the second-term course was nearly halved, the GPA gap inboth courses was reduced, and the gap in passing rate for the second-term course was eliminated.First-generation students who took the first-term course in the flipped format experienced a final exam scoreboost in the second-term course regardless of repeater status. While most students responded positively tothe flipped course structure, repeating students held a stronger preference for the flipped format. Thesefindings provide guidance on how to create courses that promote equity, access and retention of diversestudents in STEM. @font-face {font-family:Century; panose-1:2 4 6 4 5 5 5 2 3 4; mso-font-charset:0;mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;}@font-face{font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman;mso-font-pitch:variable; mso-font-signature:-536870145 1107305727 0 0 415 0;}@font-face{font-family:Cambria; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman;mso-font-pitch:variable; mso-font-signature:-536870145 1073743103 0 0 415 0;}@font-face{font-family:"Bookman Old Style"; panose-1:2 5 6 4 5 5 5 2 2 4; mso-font-charset:0;mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;}p.MsoNormal,li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in;line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Arial",sans-serif;mso-fareast-font-family:Arial; mso-ansi-language:EN;}p.JCEFlushBody, li.JCEFlushBody, div.JCEFlushBody{mso-style-name:"JCE FlushBody"; mso-style-unhide:no; mso-style-qformat:yes; mso-style-next:Normal;margin:0in; line-height:200%; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Bookman OldStyle",serif; mso-fareast-font-family:"Times New

http://doi.org/10.26434/chemrxiv.9775910.v4

https://chemrxiv.org/authors/Renee_Link/6890480

Roman"; mso-bidi-font-family:Century;}.MsoChpDefault {mso-style-type:export-only; mso-default-props:yes;font-size:10.0pt; mso-ansi-font-size:10.0pt; mso-bidi-font-size:10.0pt; font-family:"Cambria",serif;mso-ascii-font-family:Cambria; mso-fareast-font-family:Cambria;mso-hansi-font-family:Cambria;}div.WordSection1 {page:WordSection1;}@font-face {font-family:Century; panose-1:2 4 6 4 5 5 5 2 3 4; mso-font-charset:0;mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;}@font-face{font-family:"Cambria Math"; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman;mso-font-pitch:variable; mso-font-signature:-536870145 1107305727 0 0 415 0;}@font-face{font-family:Cambria; panose-1:2 4 5 3 5 4 6 3 2 4; mso-font-charset:0; mso-generic-font-family:roman;mso-font-pitch:variable; mso-font-signature:-536870145 1073743103 0 0 415 0;}@font-face{font-family:"Bookman Old Style"; panose-1:2 5 6 4 5 5 5 2 2 4; mso-font-charset:0;mso-generic-font-family:roman; mso-font-pitch:variable; mso-font-signature:647 0 0 0 159 0;}p.MsoNormal,li.MsoNormal, div.MsoNormal {mso-style-unhide:no; mso-style-qformat:yes; mso-style-parent:""; margin:0in;line-height:115%; mso-pagination:widow-orphan; font-size:11.0pt; font-family:"Arial",sans-serif;mso-fareast-font-family:Arial; mso-ansi-language:EN;}p.JCEFlushBody, li.JCEFlushBody, div.JCEFlushBody{mso-style-name:"JCE FlushBody"; mso-style-unhide:no; mso-style-qformat:yes; mso-style-next:Normal;margin:0in; line-height:200%; mso-pagination:widow-orphan; font-size:10.0pt; font-family:"Bookman OldStyle",serif; mso-fareast-font-family:"Times New Roman"; mso-bidi-font-family:Century;}.MsoChpDefault{mso-style-type:export-only; mso-default-props:yes; font-size:10.0pt; mso-ansi-font-size:10.0pt;mso-bidi-font-size:10.0pt; font-family:"Cambria",serif; mso-ascii-font-family:Cambria;mso-fareast-font-family:Cambria; mso-hansi-font-family:Cambria;}div.WordSection1 {page:WordSection1;}

File list (3)

download fileview on ChemRxivGetting Students Back on Track Survey Analysis.pdf (119.68 KiB)

download fileview on ChemRxivCHEMRXIV-REV Getting Students Back on Track Statis... (178.53 KiB)

download fileview on ChemRxivCHEMRXIV - Getting Students Back on Track Rev 2.4.202... (1.14 MiB)

https://chemrxiv.org/ndownloader/files/25528814

https://chemrxiv.org/articles/preprint/Getting_Students_Back_on_Track_Persistent_Effects_of_Flipping_Accelerated_Organic_Chemistry_on_Student_Achievement_Study_Strategies_and_Perceptions_of_Instruction/9775910/4?file=25528814





Getting Students Back on Track Survey Analysis

Year 4 Single Survey

• A single postclass survey was conducted• Likertscale items are on a 15 scale Results from this survey are presented in the text of themanuscript.

Survey Questions and Abbreviations

Code Question

readbf_num On average, I read the appropriate textbooksection (Likert neveralways)

watchbf I always watched the assigned podcasts beforeclass.

watchnotes When I watched the class podcasts I took notes.watchq When I watched the class podcasts I wrote down

questions.watchdistract When I watched the class podcasts I also did

other thing not related to class.watchbenefit I think that having class podcasts available

benefited me as a student.vidrepeat_num On average I watched each class podcast.participate I fully participated in class activities.actbenefit I think that inclass activities benefited me as a

student.enjoyothers I enjoyed working with other students during

class.discattend_num I attended discussion sections (Likert

neveralways)diccomplt_num I completed discussion problems during

discussion sections (Likert neveralways)discbenefit I think that discussion sections benefited me as a

student.hwbenefit I think that the Sapling homework benefited me as

a student.hwcorrelate I feel that the problems on Sapling correlated with

what material I learned in each chapter.flipprefer I prefer this classroom format to a traditional

lecture format.fliprecommend I would recommend an inverted class to my

friends who need to take organic chemistry.flipeffective I think listening to lecture outside of class and

working on problem solving in class is an effectiveway to learn.

1

Code Question

moreflip I would prefer to take more science classes thatuse this type of class format.

toomuchwork There is too much work outside of class time inthis course as compared to other science courses.

Summary Data

Summary Statistics Full Class

skim_variable n_missing complete_rate ordered n_unique top_counts

repeater 0 1.0000000 FALSE 2 Yes: 120, No: 78gender 2 0.9898990 FALSE 2 Fem: 126, Mal: 70level 2 0.9898990 FALSE 4 jun: 108, sen: 44, sop: 37, alr: 7jobhrs 43 0.7828283 FALSE 3 05: 114, 15: 22, >20: 19, 61: 0

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100

watchbf 1 0.9949495 3.984772 1.1757636 1 3.00 4.0 5.00 5watchnotes 0 1.0000000 3.964647 1.2918183 1 3.00 4.5 5.00 5watchq 0 1.0000000 2.888889 1.2289973 1 2.00 3.0 4.00 5watchdistract 0 1.0000000 2.323232 1.1735519 1 1.00 2.0 3.00 5watchbenefit 0 1.0000000 4.363636 0.8950461 1 4.00 5.0 5.00 5

participate 0 1.0000000 3.808081 0.9088037 1 3.00 4.0 4.00 5actbenefit 0 1.0000000 4.080808 0.8511177 1 4.00 4.0 5.00 5enjoyothers 0 1.0000000 3.505050 0.8827016 1 3.00 3.0 4.00 5discbenefit 0 1.0000000 3.893939 1.1192200 1 3.00 4.0 5.00 5hwbenefit 1 0.9949495 4.101523 0.8747289 1 4.00 4.0 5.00 5

hwcorrelate 2 0.9898990 4.102041 0.8226261 1 4.00 4.0 5.00 5msgbrdbenefit 0 1.0000000 2.646465 0.9269034 1 2.00 3.0 3.00 5flipprefer 3 0.9848485 3.589744 1.2946748 1 3.00 4.0 5.00 5fliprecommend 0 1.0000000 4.045454 1.0629690 1 3.25 4.0 5.00 5flipeffective 0 1.0000000 4.161616 1.0393972 1 4.00 4.0 5.00 5

moreflip 0 1.0000000 4.005051 1.0781516 1 3.00 4.0 5.00 5toomuchwork 1 0.9949495 3.040609 1.0918883 1 2.00 3.0 4.00 5readbf_num 0 1.0000000 3.646465 1.2609974 1 3.00 4.0 5.00 5discattend_num 0 1.0000000 3.787879 1.2846041 1 3.00 4.0 5.00 5disccomplt_num 2 0.9898990 3.918367 1.2903863 1 3.00 4.0 5.00 5

msgbrduse_num 0 1.0000000 1.505051 0.7853188 1 1.00 1.0 2.00 5vidrepeat_num 0 1.0000000 2.101010 1.3175500 1 1.00 2.0 2.75 5

2

Summary Statistics by Repeater Status

skim_variable repeater n_missing complete_rate ordered n_unique top_counts

gender Yes 2 0.9833333 FALSE 2 Fem: 82, Mal: 36gender No 0 1.0000000 FALSE 2 Fem: 44, Mal: 34level Yes 1 0.9916667 FALSE 4 jun: 79, sen: 29, sop: 7, alr: 4level No 1 0.9871795 FALSE 4 sop: 30, jun: 29, sen: 15, alr: 3jobhrs Yes 26 0.7833333 FALSE 3 05: 68, 15: 16, >20: 10, 61: 0

jobhrs No 17 0.7820513 FALSE 3 05: 46, >20: 9, 15: 6, 61: 0

3

skim_variable repeater n_missing complete_rate mean sd p0 p25 p50 p75 p100

watchbf Yes 1 0.9916667 3.966387 1.1998077 1 3.00 4.0 5.00 5watchbf No 0 1.0000000 4.012821 1.1452169 1 4.00 4.0 5.00 5watchnotes Yes 0 1.0000000 3.900000 1.3559703 1 3.00 4.5 5.00 5watchnotes No 0 1.0000000 4.064103 1.1880328 1 3.25 4.5 5.00 5watchq Yes 0 1.0000000 2.933333 1.3011523 1 2.00 3.0 4.00 5

watchq No 0 1.0000000 2.820513 1.1135947 1 2.00 3.0 4.00 5watchdistract Yes 0 1.0000000 2.416667 1.1709605 1 1.00 2.0 3.00 5watchdistract No 0 1.0000000 2.179487 1.1704543 1 1.00 2.0 2.75 5watchbenefit Yes 0 1.0000000 4.400000 0.8638394 1 4.00 5.0 5.00 5watchbenefit No 0 1.0000000 4.307692 0.9439856 1 4.00 5.0 5.00 5

participate Yes 0 1.0000000 3.916667 0.8258770 1 4.00 4.0 4.00 5participate No 0 1.0000000 3.641026 1.0061416 1 3.00 4.0 4.00 5actbenefit Yes 0 1.0000000 4.216667 0.7239404 1 4.00 4.0 5.00 5actbenefit No 0 1.0000000 3.871795 0.9850700 1 3.00 4.0 5.00 5enjoyothers Yes 0 1.0000000 3.508333 0.9166603 1 3.00 3.5 4.00 5

enjoyothers No 0 1.0000000 3.500000 0.8335498 1 3.00 3.0 4.00 5discbenefit Yes 0 1.0000000 3.866667 1.1662865 1 3.00 4.0 5.00 5discbenefit No 0 1.0000000 3.935897 1.0486818 1 4.00 4.0 5.00 5hwbenefit Yes 0 1.0000000 4.125000 0.8654187 1 4.00 4.0 5.00 5hwbenefit No 1 0.9871795 4.064935 0.8935097 1 4.00 4.0 5.00 5

hwcorrelate Yes 2 0.9833333 4.169492 0.8091896 1 4.00 4.0 5.00 5hwcorrelate No 0 1.0000000 4.000000 0.8374358 2 4.00 4.0 5.00 5msgbrdbenefit Yes 0 1.0000000 2.666667 0.9378442 1 2.00 3.0 3.00 5msgbrdbenefit No 0 1.0000000 2.615385 0.9149660 1 2.00 3.0 3.00 5flipprefer Yes 3 0.9750000 3.658120 1.2943005 1 3.00 4.0 5.00 5

flipprefer No 0 1.0000000 3.487179 1.2967851 1 3.00 4.0 4.75 5fliprecommend Yes 0 1.0000000 4.258333 0.9120651 1 4.00 4.5 5.00 5fliprecommend No 0 1.0000000 3.717949 1.1941137 1 3.00 4.0 5.00 5flipeffective Yes 0 1.0000000 4.416667 0.8359902 1 4.00 5.0 5.00 5flipeffective No 0 1.0000000 3.769231 1.1943925 1 3.00 4.0 5.00 5

moreflip Yes 0 1.0000000 4.208333 0.9157431 1 4.00 4.0 5.00 5moreflip No 0 1.0000000 3.692308 1.2306443 1 3.00 4.0 5.00 5toomuchwork Yes 1 0.9916667 2.983193 1.0332046 1 2.00 3.0 4.00 5toomuchwork No 0 1.0000000 3.128205 1.1772627 1 2.00 3.0 4.00 5readbf_num Yes 0 1.0000000 3.591667 1.2800724 1 3.00 4.0 5.00 5

readbf_num No 0 1.0000000 3.730769 1.2344942 1 3.00 4.0 5.00 5discattend_num Yes 0 1.0000000 3.800000 1.2873006 1 3.00 4.0 5.00 5discattend_num No 0 1.0000000 3.769231 1.2885417 1 3.00 4.0 5.00 5disccomplt_num Yes 2 0.9833333 3.991525 1.3428873 1 4.00 4.0 5.00 5disccomplt_num No 0 1.0000000 3.807692 1.2066665 1 3.00 4.0 5.00 5

msgbrduse_num Yes 0 1.0000000 1.541667 0.8391255 1 1.00 1.0 2.00 5msgbrduse_num No 0 1.0000000 1.448718 0.6959519 1 1.00 1.0 2.00 3vidrepeat_num Yes 0 1.0000000 2.216667 1.3852969 1 1.00 2.0 3.00 5vidrepeat_num No 0 1.0000000 1.923077 1.1927185 1 1.00 2.0 2.00 5

Statistical Signficance Tests for Repeater vs NonRepeater

We used Wilcoxon rank sum tests with continuity corrections (twotailed) to identify any items that hadstatistically significant differences between NonRepeaters and Repeaters, and we controlled the overallType I error rate to be 0.05 using a Bonferroni correction (𝛼*= 𝛼/m = 0.05/12 = 0.0042).

4

question statistic p.value method alternative

watchnotes 4487.0 0.5989124 Wilcoxon rank sum test with continuity correction two.sidedwatchq 4902.0 0.5635273 Wilcoxon rank sum test with continuity correction two.sidedwatchdistract 5278.0 0.1148654 Wilcoxon rank sum test with continuity correction two.sidedwatchbenefit 4898.0 0.5356559 Wilcoxon rank sum test with continuity correction two.sidedparticipate 5371.5 0.0563700 Wilcoxon rank sum test with continuity correction two.sided

actbenefit 5539.5 0.0173095 Wilcoxon rank sum test with continuity correction two.sidedenjoyothers 4738.5 0.8754769 Wilcoxon rank sum test with continuity correction two.sidedflipprefer 4931.5 0.3236356 Wilcoxon rank sum test with continuity correction two.sidedfliprecommend 5898.5 0.0010183 Wilcoxon rank sum test with continuity correction two.sidedflipeffective 6187.0 0.0000353 Wilcoxon rank sum test with continuity correction two.sided

moreflip 5769.5 0.0034530 Wilcoxon rank sum test with continuity correction two.sidedtoomuchwork 4330.5 0.4115719 Wilcoxon rank sum test with continuity correction two.sided

5

5%

5%

5%

5%

9%

9%

17%

10%

10%

11%

17%

20%

16%

10%

42%

35%

64%

88%

83%

82%

82%

81%

75%

74%

72%

72%

71%

71%

62%

56%

49%

35%

34%

19%

7%

12%

13%

14%

11%

16%

9%

18%

18%

18%

12%

18%

29%

41%

23%

31%

17%

I always watched the assigned podcasts beforeclass.

When I watched the class podcasts I took notes.

When I watched the class podcasts I wrote downquestions.

When I watched the class podcasts I also didother thing not related to class.

I think that having class podcasts availablebenefited me as a student.

I fully participated in class activities.

I think that in−class activities benefited me asa student.

I enjoyed working with other students duringclass.

I think that discussion sections benefited me asa student.

I think that the Sapling homework benefited me asa student.

I feel that the problems on Sapling correlatedwith what material I learned in each chapter.

I prefer this classroom format to a traditionallecture format.

I would recommend a inverted class to my friendswho need to take organic chemistry.

I think listening to lecture outside of class andworking on problem solving in class is an

effective way to learn.

I would prefer to take more science classes thatuse this type of class format.

There is too much work outside of class time inthis course as compared to other science courses.

On average, I read the appropriate textbooksection (never−always)

0 25 50 75 100Percentage

Response 1 2 3 4 5

Full Class Summary

6

5%

6%

8%

17%

9%

13%

10%

16%

17%

17%

17%

17%

25%

8%

34%

47%

74%

84%

84%

81%

77%

75%

74%

73%

65%

64%

62%

61%

60%

57%

47%

39%

32%

17%

10%

9%

12%

6%

16%

13%

17%

19%

19%

21%

22%

23%

18%

45%

27%

21%

9%

I always watched the assigned podcasts beforeclass.

When I watched the class podcasts I took notes.

When I watched the class podcasts I wrote downquestions.

When I watched the class podcasts I also didother thing not related to class.

I think that having class podcasts availablebenefited me as a student.

I fully participated in class activities.

I think that in−class activities benefited me asa student.

I enjoyed working with other students duringclass.

I think that discussion sections benefited me asa student.

I think that the Sapling homework benefited me asa student.

I feel that the problems on Sapling correlatedwith what material I learned in each chapter.

I prefer this classroom format to a traditionallecture format.

I would recommend a inverted class to my friendswho need to take organic chemistry.

I think listening to lecture outside of class andworking on problem solving in class is an

effective way to learn.

I would prefer to take more science classes thatuse this type of class format.

There is too much work outside of class time inthis course as compared to other science courses.

On average, I read the appropriate textbooksection (never−always)


Response 1 2 3 4 5

2012 Non−Repeating Students' Survey Responses

7

Year 5 Pre and Postclass Survey – Combined Data

For Year 5, we used a 6point scale and administered a pre and postsurvey. The survey items weremodified to obtain more detailed information and see if there were any significant changes from the beginningof the class to the end of the class. Similar to results for Year 4, we found the majority of students in Year 5preferred the flipped format.

Similar to results for Year 4, we found the majority of students in Year 5 preferred the flipped format. Repeaters’ preference for flipped instructional practices were higher than NonRepeaters, but not statisticallydifferent (Table 9). We used Wilcoxon rank sum tests with continuity corrections (twotailed) to identify anyitems that had statistically significant differences between NonRepeaters and Repeaters, and we controlledthe overall Type I error rate to be 0.05 using a Bonferroni correction (𝛼*= 𝛼/m = 0.05/19 = 0.0026). Theseitems focused on study habits, motivation and predicted course grade. While five items were found to bestatistically significant, this may be due to a change in the wording of the question between pre and postsurvey, confounding these results.

Survey Questions and Abbreviations

PreClass Survey

Code Question

bfhrs1 In a regular 10week quarter, before learninganything from lectures, how many hours per weekon average do you spend studying for a typicalscience or math course in advance? (numericresponse paired)

afthrs1 In a regular 10week quarter, after attendinglectures, how many hours per week on averagedo you spend studying for a typical science ormath course (please do not include lab ordiscussion sessions)? (numeric response paired)

qinclass1 I asked questions in class. (paired)qtta1 I asked the instructor or the TA for help after

class. (paired)qstudents1 I asked other students for help. (paired)qno1 I preferred not to ask others for help, even when I

couldn’t figure things out. (paired)notes1 I take notes to summarize important points when

learning new material. (paired)question1 I write down questions I have in mind when

learning. (paired)interest1 I am very interested in the content area of this

course. (paired)selfeff1 Question goes hereutility1 Beyond this summer session, this course will still

be very useful to me. (paired)mot1 I am highly motivated to do well in this course.

(paired)

8

PostClass Survey

Code Question

bfhrs2 On average, I spent _____ hours each week intotal learning course material and doing assignedtasks before class. (numeric response paired)

bfvhrs Of that total, I spent about _____ hours watchingpodcasts before class.(numeric response)

bfbhrs Of that total, I spent about _____ hours readingthe textbook before class

afthrs2 By contrast, I spent about _____ hours each weeklearning course material and doing assignedtasks after class (numeric response paired)

interest2 I am very interested in the content area of thiscourse. (paired)

mot2 I am highly motivated to do well in this course.(paired)

utility2 Beyond this summer session, this course will stillbe very useful to me. (paired)

seleff2 I am very confident that I will do well in thiscourse. (paired)

notes2 I frequently took notes when learning newmaterial before class. (paired)

question2 I frequently wrote down questions I had whenlearning new material before class. (paired)

uc_q Knowing exactly what I didn’t understand mademe more likely to ask questions.

lbl_uc Learning before class helped me find out things Ididn’t understand.

l_answq The inclass activities often answered questions Ihad from learning before class.

classengage I was intellectually engaged in class throughoutthe class.

fnotes I frequently summarized important points in noteswhen learning before class.

fdiscuss I frequently attended discussion sections.hybridprefer I prefer this learnbeforeclass format to a

“traditional lecture” format.hybridmore I would like to take more science classes using

this type of class format.discprob I often completed discussion problems during

discussion sections.qinclass2 I asked questions directly during class. (paired)qtta2 I asked the instructor or the TA for help after

class. (paired)qstudents2 I asked other students for help. (paired)qno2 I preferred not to ask others for help, even when I

couldn’t figure things out. (paired)hvideo Having class podcasts available (Likert very

ineffective very effective)hsapling Doing Sapling homework (Likert very ineffective

very effective)

9

Code Question

hclassacts Participating in various inclass activities (Likertvery ineffective very effective)

hdiscussion Attending discussion sections (Likert veryineffective very effective)

htogether Working with other students during class (Likertvery ineffective very effective)

hlbl Learning before class (Likert very ineffective very effective)

hcourse This course as a whole (Likert very ineffective very effective)

Summary Statistics

Summary Statistics Full Class

skim_variable n_missing complete_rate ordered n_unique top_counts

repeater 0 1.0000000 FALSE 2 No: 102, Yes: 96lvl 0 1.0000000 FALSE 4 JR: 81, SR: 54, SO: 50, FR: 13d51B 24 0.8787879 FALSE 3 Yes: 106, No: 56, Don: 12why 41 0.7929293 FALSE 2 I a: 106, To : 51bfvcompre 19 0.9040404 FALSE 5 61%: 71, 81%: 63, 41%: 32, 21%: 11

bfbcompre 19 0.9040404 FALSE 5 61%: 58, 81%: 50, 41%: 31, 0% : 23

10

skim_variable n_missing complete_rate mean sd p0 p25 p50 p75 p100

units 24 0.8787879 4.8850575 3.7610732 0 4 4.0 6 19bfhrs1 24 0.8787879 3.9942529 4.7780170 0 1 2.5 5 30afthrs1 24 0.8787879 8.5660920 7.7250400 0 4 6.0 10 55qinclass1 24 0.8787879 1.8620690 0.9145252 1 1 2.0 2 6qtta1 24 0.8787879 2.8160920 1.2167305 1 2 3.0 3 6

qstudents1 24 0.8787879 3.8563218 1.2525305 1 3 4.0 5 6qno1 24 0.8787879 2.6494253 1.3334621 1 1 3.0 3 6notes1 24 0.8787879 4.5862069 1.1434834 1 4 5.0 6 6question1 24 0.8787879 3.0574713 1.3243502 1 2 3.0 4 6interest1 24 0.8787879 4.1666667 1.2025182 1 4 4.0 5 6

selfeff1 24 0.8787879 4.0344828 1.0906675 1 4 4.0 5 6utility1 24 0.8787879 4.5517241 1.1997940 1 4 5.0 5 6mot1 24 0.8787879 5.3448276 0.9227702 2 5 6.0 6 6bfhrs2 19 0.9040404 8.1452514 7.2510067 1 4 6.0 10 48bfvhrs 19 0.9040404 2.6888268 2.4705657 0 1 2.0 3 20

afthrs2 19 0.9040404 6.3519553 9.6385016 0 2 4.0 8 99otherhrs 19 0.9040404 13.7458101 12.7756537 0 5 10.0 20 80tutorhrs 19 0.9040404 0.5932961 1.3345705 0 0 0.0 0 6interest2 19 0.9040404 4.1787709 1.2230909 1 4 4.0 5 6mot2 19 0.9040404 4.8603352 1.1886835 1 4 5.0 6 6

utility2 19 0.9040404 4.6592179 1.3243873 1 4 5.0 6 6selfeff2 19 0.9040404 3.7039106 1.2836641 1 3 4.0 5 6videoclear 19 0.9040404 4.4357542 1.1756212 1 4 4.0 5 6notes2 19 0.9040404 4.5474860 1.3623063 1 4 5.0 6 6question2 19 0.9040404 3.0837989 1.3977169 1 2 3.0 4 6

classclear 19 0.9040404 3.7039106 1.2660370 1 3 4.0 4 6classconfuse 19 0.9040404 3.3519553 1.3674113 1 2 3.0 4 6uc_q 19 0.9040404 3.7430168 1.4303664 1 3 4.0 5 6lbl_uc 19 0.9040404 4.4469274 1.2230139 1 4 5.0 5 6l_answq 19 0.9040404 3.9832402 1.2607955 1 3 4.0 5 6

classengage 19 0.9040404 3.8603352 1.3688106 1 3 4.0 5 6fnotes 19 0.9040404 3.8491620 1.4819653 1 3 4.0 5 6fdiscuss 19 0.9040404 3.6815642 1.8736031 1 2 4.0 6 6hybridprefer 19 0.9040404 4.0279330 1.7302015 1 3 4.0 6 6fmsboard 19 0.9040404 1.7541899 1.2343045 1 1 1.0 2 6

hybridmore 19 0.9040404 3.9553073 1.7379480 1 3 4.0 6 6discprob 19 0.9040404 3.4916201 1.7812025 1 2 4.0 5 6qinclass2 19 0.9040404 2.0893855 1.3544969 1 1 2.0 3 6qtta2 19 0.9040404 2.4916201 1.6671604 1 1 2.0 4 6qtutor2 19 0.9040404 2.1005587 1.5217338 1 1 1.0 3 6

qmoretutor2 19 0.9040404 1.8659218 1.5154302 1 1 1.0 2 6qstudents2 19 0.9040404 3.5530726 1.6322691 1 2 4.0 5 6qno2 19 0.9040404 3.2234637 1.5597851 1 2 3.0 4 6hvideo 19 0.9040404 4.9608939 1.2149034 1 4 5.0 6 6hsapling 19 0.9040404 4.6592179 1.1568368 2 4 5.0 6 6

hclassacts 19 0.9040404 4.0055866 1.4436891 1 3 4.0 5 6hdiscussion 19 0.9040404 3.7541899 1.6304414 1 3 4.0 5 6hmsboard 19 0.9040404 2.2625698 1.3421834 1 1 2.0 3 6htogether 19 0.9040404 3.4860335 1.4891274 1 2 3.0 5 6hlbl 19 0.9040404 4.7318436 1.2567313 1 4 5.0 6 6

hcourse 19 0.9040404 4.4692737 1.0979578 1 4 5.0 5 6

11

By Repeater Status

skim_variable repeater n_missing complete_rate ordered n_unique top_counts

lvl Yes 0 1.0000000 FALSE 3 JR: 42, SR: 35, SO: 19, FR: 0lvl No 0 1.0000000 FALSE 4 JR: 39, SO: 31, SR: 19, FR: 13d51B Yes 8 0.9166667 FALSE 3 Yes: 58, No: 22, Don: 8d51B No 16 0.8431373 FALSE 3 Yes: 48, No: 34, Don: 4why Yes 10 0.8958333 FALSE 2 I a: 80, To : 6

why No 31 0.6960784 FALSE 2 To : 45, I a: 26bfvcompre Yes 8 0.9166667 FALSE 5 61%: 38, 81%: 28, 41%: 13, 21%: 8bfvcompre No 11 0.8921569 FALSE 5 81%: 35, 61%: 33, 41%: 19, 21%: 3bfbcompre Yes 8 0.9166667 FALSE 5 61%: 26, 81%: 25, 41%: 19, 0% : 11bfbcompre No 11 0.8921569 FALSE 5 61%: 32, 81%: 25, 0% : 12, 41%: 12


units Yes 8 0.9166667 4.8522727 3.3845143 0 4.000 4.0 4.25 19units No 16 0.8431373 4.9186047 4.1308456 0 2.000 4.0 7.50 19bfhrs1 Yes 8 0.9166667 3.7670455 3.9949313 0 1.875 3.0 5.00 25bfhrs1 No 16 0.8431373 4.2267442 5.4791890 0 1.000 2.0 5.00 30afthrs1 Yes 8 0.9166667 7.3068182 4.7242538 1 4.000 6.0 10.00 20

afthrs1 No 16 0.8431373 9.8546512 9.7618303 0 4.000 7.0 10.00 55qinclass1 Yes 8 0.9166667 1.8181818 0.9039562 1 1.000 2.0 2.00 6qinclass1 No 16 0.8431373 1.9069767 0.9283497 1 1.000 2.0 2.00 6qtta1 Yes 8 0.9166667 2.5909091 1.0789135 1 2.000 3.0 3.00 6qtta1 No 16 0.8431373 3.0465116 1.3097550 1 2.000 3.0 4.00 6

qstudents1 Yes 8 0.9166667 3.7045455 1.2425326 1 3.000 4.0 5.00 6qstudents1 No 16 0.8431373 4.0116279 1.2508274 1 3.000 4.0 5.00 6qno1 Yes 8 0.9166667 2.7159091 1.2680772 1 2.000 3.0 3.00 6qno1 No 16 0.8431373 2.5813953 1.4013868 1 1.000 3.0 3.00 6notes1 Yes 8 0.9166667 4.5227273 1.1242088 2 4.000 5.0 5.00 6

notes1 No 16 0.8431373 4.6511628 1.1658619 1 4.000 5.0 6.00 6question1 Yes 8 0.9166667 2.9659091 1.3514476 1 2.000 3.0 3.00 6question1 No 16 0.8431373 3.1511628 1.2972137 1 2.000 3.0 4.00 6interest1 Yes 8 0.9166667 4.0000000 1.1646123 1 4.000 4.0 5.00 6interest1 No 16 0.8431373 4.3372093 1.2234038 1 4.000 4.5 5.00 6

selfeff1 Yes 8 0.9166667 3.9090909 1.1510751 1 3.000 4.0 5.00 6selfeff1 No 16 0.8431373 4.1627907 1.0158794 1 4.000 4.0 5.00 6utility1 Yes 8 0.9166667 4.3181818 1.1699833 1 4.000 4.0 5.00 6utility1 No 16 0.8431373 4.7906977 1.1893278 1 4.000 5.0 6.00 6mot1 Yes 8 0.9166667 5.3068182 0.9632077 2 5.000 6.0 6.00 6

mot1 No 16 0.8431373 5.3837209 0.8834239 2 5.000 6.0 6.00 6bfhrs2 Yes 8 0.9166667 8.1704545 7.4374350 1 3.750 6.0 10.00 35bfhrs2 No 11 0.8921569 8.1208791 7.1072501 1 4.000 6.0 10.00 48bfvhrs Yes 8 0.9166667 2.9431818 3.1150652 0 1.000 2.0 3.00 20bfvhrs No 11 0.8921569 2.4428571 1.6022306 0 1.000 2.0 3.00 8

afthrs2 Yes 8 0.9166667 6.5284091 11.6113693 0 2.000 4.0 8.00 99afthrs2 No 11 0.8921569 6.1813187 7.3039245 0 2.000 4.0 7.50 50otherhrs Yes 8 0.9166667 15.5454545 12.9897476 0 6.000 12.0 20.00 80otherhrs No 11 0.8921569 12.0054945 12.3882773 0 3.000 8.0 17.50 50tutorhrs Yes 8 0.9166667 0.5931818 1.4266588 0 0.000 0.0 0.00 6

tutorhrs No 11 0.8921569 0.5934066 1.2470233 0 0.000 0.0 0.00 4

12

(continued)


interest2 Yes 8 0.9166667 4.0568182 1.1382377 1 3.000 4.0 5.00 6interest2 No 11 0.8921569 4.2967033 1.2952435 1 4.000 4.0 5.00 6mot2 Yes 8 0.9166667 4.8750000 1.1327162 1 4.000 5.0 6.00 6mot2 No 11 0.8921569 4.8461538 1.2465337 1 4.000 5.0 6.00 6

utility2 Yes 8 0.9166667 4.5681818 1.2936223 1 4.000 5.0 6.00 6utility2 No 11 0.8921569 4.7472527 1.3547727 1 4.000 5.0 6.00 6selfeff2 Yes 8 0.9166667 3.7613636 1.2775184 1 3.000 4.0 4.25 6selfeff2 No 11 0.8921569 3.6483516 1.2942061 1 3.000 4.0 5.00 6videoclear Yes 8 0.9166667 4.3636364 1.2241048 1 4.000 4.0 5.00 6

videoclear No 11 0.8921569 4.5054945 1.1291455 1 4.000 5.0 5.00 6notes2 Yes 8 0.9166667 4.5227273 1.3976768 1 4.000 5.0 6.00 6notes2 No 11 0.8921569 4.5714286 1.3345233 1 4.000 5.0 6.00 6question2 Yes 8 0.9166667 3.0795455 1.3748278 1 2.000 3.0 4.00 6question2 No 11 0.8921569 3.0879121 1.4271055 1 2.000 3.0 4.00 6

classclear Yes 8 0.9166667 3.6931818 1.2443708 1 3.000 4.0 4.00 6classclear No 11 0.8921569 3.7142857 1.2934511 1 3.000 4.0 5.00 6classconfuse Yes 8 0.9166667 3.3522727 1.3046316 1 2.000 4.0 4.00 6classconfuse No 11 0.8921569 3.3516484 1.4327412 1 2.000 3.0 4.00 6uc_q Yes 8 0.9166667 3.7386364 1.4263072 1 3.000 4.0 5.00 6

uc_q No 11 0.8921569 3.7472527 1.4421697 1 3.000 4.0 5.00 6lbl_uc Yes 8 0.9166667 4.3636364 1.2427428 1 4.000 5.0 5.00 6lbl_uc No 11 0.8921569 4.5274725 1.2049957 2 4.000 5.0 5.50 6l_answq Yes 8 0.9166667 4.0113636 1.2176321 1 3.000 4.0 5.00 6l_answq No 11 0.8921569 3.9560440 1.3073475 1 3.000 4.0 5.00 6

classengage Yes 8 0.9166667 3.8636364 1.3058825 1 3.000 4.0 5.00 6classengage No 11 0.8921569 3.8571429 1.4342743 1 3.000 4.0 5.00 6fnotes Yes 8 0.9166667 3.7386364 1.5274825 1 3.000 4.0 5.00 6fnotes No 11 0.8921569 3.9560440 1.4369109 1 3.000 4.0 5.00 6fdiscuss Yes 8 0.9166667 3.5000000 1.7811271 1 2.000 3.0 5.00 6

fdiscuss No 11 0.8921569 3.8571429 1.9526132 1 2.000 4.0 6.00 6hybridprefer Yes 8 0.9166667 4.1477273 1.7055674 1 3.000 5.0 6.00 6hybridprefer No 11 0.8921569 3.9120879 1.7552990 1 3.000 4.0 5.50 6fmsboard Yes 8 0.9166667 1.6477273 1.1350201 1 1.000 1.0 2.00 6fmsboard No 11 0.8921569 1.8571429 1.3213749 1 1.000 1.0 2.00 6

hybridmore Yes 8 0.9166667 4.1136364 1.7182711 1 3.000 4.0 6.00 6hybridmore No 11 0.8921569 3.8021978 1.7525840 1 3.000 4.0 5.00 6discprob Yes 8 0.9166667 3.3750000 1.6972764 1 2.000 3.0 5.00 6discprob No 11 0.8921569 3.6043956 1.8611772 1 1.500 4.0 5.00 6qinclass2 Yes 8 0.9166667 1.9318182 1.2110170 1 1.000 1.0 3.00 6

qinclass2 No 11 0.8921569 2.2417582 1.4707569 1 1.000 2.0 3.00 6qtta2 Yes 8 0.9166667 2.3863636 1.5419932 1 1.000 2.0 3.00 6qtta2 No 11 0.8921569 2.5934066 1.7824953 1 1.000 2.0 4.00 6qtutor2 Yes 8 0.9166667 2.0681818 1.5222145 1 1.000 1.0 3.00 6qtutor2 No 11 0.8921569 2.1318681 1.5290432 1 1.000 1.0 3.00 6

qmoretutor2 Yes 8 0.9166667 1.7613636 1.4222721 1 1.000 1.0 2.00 6qmoretutor2 No 11 0.8921569 1.9670330 1.6017390 1 1.000 1.0 2.00 6qstudents2 Yes 8 0.9166667 3.5227273 1.5311124 1 2.000 4.0 5.00 6qstudents2 No 11 0.8921569 3.5824176 1.7324737 1 2.000 4.0 5.00 6qno2 Yes 8 0.9166667 3.1931818 1.5226863 1 2.000 3.0 5.00 6

qno2 No 11 0.8921569 3.2527473 1.6027297 1 2.000 3.0 4.00 6hvideo Yes 8 0.9166667 5.0568182 1.1582582 1 4.000 5.0 6.00 6

13

(continued)


hvideo No 11 0.8921569 4.8681319 1.2667438 1 4.000 5.0 6.00 6hsapling Yes 8 0.9166667 4.8522727 1.0884955 2 4.000 5.0 6.00 6hsapling No 11 0.8921569 4.4725275 1.1957393 2 4.000 5.0 5.00 6

hclassacts Yes 8 0.9166667 4.0227273 1.4698289 1 3.000 4.0 5.00 6hclassacts No 11 0.8921569 3.9890110 1.4259072 1 3.000 4.0 5.00 6hdiscussion Yes 8 0.9166667 3.7386364 1.6009646 1 3.000 4.0 5.00 6hdiscussion No 11 0.8921569 3.7692308 1.6671794 1 3.000 4.0 5.00 6hmsboard Yes 8 0.9166667 2.4090909 1.3699263 1 1.000 2.0 3.00 6

hmsboard No 11 0.8921569 2.1208791 1.3066936 1 1.000 2.0 3.00 6htogether Yes 8 0.9166667 3.3409091 1.3803745 1 2.000 3.0 4.00 6htogether No 11 0.8921569 3.6263736 1.5821810 1 2.500 4.0 5.00 6hlbl Yes 8 0.9166667 4.8181818 1.1501669 1 4.000 5.0 6.00 6hlbl No 11 0.8921569 4.6483516 1.3529690 1 4.000 5.0 6.00 6

hcourse Yes 8 0.9166667 4.4659091 0.9820090 2 4.000 5.0 5.00 6hcourse No 11 0.8921569 4.4725275 1.2049957 1 4.000 5.0 5.00 6

Statistical Significance Tests for Paired Questions (not separated by repeater status)


bfhrs 10158.5 0.0000000 Wilcoxon signed rank test with continuity correction two.sidedafthrs 2756.0 0.0000001 Wilcoxon signed rank test with continuity correction two.sidedqinclass 3233.5 0.0118809 Wilcoxon signed rank test with continuity correction two.sidedqtta 3094.5 0.0471794 Wilcoxon signed rank test with continuity correction two.sidedqstudents 3061.0 0.0529623 Wilcoxon signed rank test with continuity correction two.sided

qno 4683.5 0.0000190 Wilcoxon signed rank test with continuity correction two.sidednotes 2971.5 0.6812913 Wilcoxon signed rank test with continuity correction two.sidedinterest 2336.0 0.6817908 Wilcoxon signed rank test with continuity correction two.sidedselfeff 2100.5 0.0012859 Wilcoxon signed rank test with continuity correction two.sidedutility 2515.0 0.3653614 Wilcoxon signed rank test with continuity correction two.sided

mot 678.0 0.0000003 Wilcoxon signed rank test with continuity correction two.sided

Statistical Signficance Tests for Repeater vs NonRepeater


hvideo 4313.5 0.3424606 Wilcoxon rank sum test with continuity correction two.sidedhclassacts 4105.0 0.7672130 Wilcoxon rank sum test with continuity correction two.sidedhdiscussion 3925.5 0.8188848 Wilcoxon rank sum test with continuity correction two.sidedhtogether 3554.0 0.1857354 Wilcoxon rank sum test with continuity correction two.sidedhlbl 4182.0 0.5938019 Wilcoxon rank sum test with continuity correction two.sided

hcourse 3846.0 0.6355781 Wilcoxon rank sum test with continuity correction two.sidedhybridprefer 4313.5 0.3632205 Wilcoxon rank sum test with continuity correction two.sidedhybridmore 4419.5 0.2222297 Wilcoxon rank sum test with continuity correction two.sided

14

12%

13%

15%

16%

17%

19%

22%

23%

25%

32%

35%

36%

37%

40%

40%

43%

44%

47%

49%

54%

58%

63%

74%

85%

88%

87%

85%

84%

83%

81%

78%

77%

75%

68%

65%

64%

63%

60%

60%

57%

56%

53%

51%

46%

42%

37%

26%

15%

I am very interested in the content area of thiscourse.

I am highly motivated to do well in this course.

Beyond this summer session, this course willstill be very useful to me.

I am very confident that I will do well in thiscourse.

I frequently took notes when learning newmaterial before class.

I frequently wrote down questions I had whenlearning new material before class.

Knowing exactly what I didn...t understand made memore likely to ask questions.

Learning before class helped me find out things Ididn...t understand.

The in−class activities often answered questionsI had from learning before class.

I was intellectually engaged in class throughoutthe class.

I frequently summarized important points in noteswhen learning before class.

I prefer this learn−before−class format to a...traditional lecture... format.

I would like to take more science classes usingthis type of class format.

I asked questions directly during class.

I asked the instructor or the TA for help afterclass.

I asked other students for help.

I preferred not to ask others for help, even whenI couldn...t figure things out.

Having class podcasts available

Doing Sapling homework

Participating in various in−class activities

Attending discussion sections

Working with other students during class

Learning before class

This course as a whole


Response1

2

3

4

5

6

2013 Post−Survey Full Class Summary

15

14%

15%

15%

20%

20%

21%

23%

23%

23%

31%

37%

38%

40%

41%

41%

42%

44%

46%

48%

49%

57%

65%

70%

80%

86%

85%

85%

80%

80%

79%

77%

77%

77%

69%

63%

62%

60%

59%

59%

58%

56%

54%

52%

51%

43%

35%

30%

20%


























Response1

2

3

4

5

6

2013 Non−Repeating Students' Survey Responses

16

9%

10%

10%

13%

15%

19%

20%

24%

28%

30%

33%

33%

34%

36%

41%

45%

47%

48%

49%

59%

60%

61%

78%

91%

91%

90%

90%

88%

85%

81%

80%

76%

72%

70%

67%

67%

66%

64%

59%

55%

53%

52%

51%

41%

40%

39%

22%

9%


























Response1

2

3

4

5

6

2013 Repeating Students' Survey Responses

17

download fileview on ChemRxivGetting Students Back on Track Survey Analysis.pdf (119.68 KiB)



2/4/21 Page 1 of 6

Linear Mixed Model Methods and Results LINEAR MIXED EFFECTS MODELS

Linear mixed effects models were fit to the data to account for the correlation of students nested

within a class. Separate models were fit for the final exam and the midterms. All models are provided

below with a brief description. A description of using linear mixed effects models is provided here for 5

readers who might not be familiar with this method.

The linear mixed model is given by:

(1)

where Yij is the response of the jth student of class i (i = 1,...,3, j = 1,…,ni), ni is the size of the class

i, xij is the covariate vector of the jth student of class i for the fixed effects (flipped classroom and the 10

student repeated Organic Chemistry I, flipped classroom and the student is a non-repeater of Organic

Chemistry I, traditional classroom and the student repeated Organic Chemistry I versus the reference

group of a traditional classroom and the student is a non-repeater of Organic Chemistry I), β is the

fixed effects parameter, uij is the covariate vector of the jth student of class i for the random effects, γi

is the random effect parameter, εij is the random error associated with the jth student of class i, and εi 15

is the error vector of class i. The model assumptions are 1) the random effects parameter follows a

Gaussian (normal) distribution with mean zero and covariance matrix D, 2) the random error for class

i follows a Gaussian distribution with mean zero and covariance matrix Σi, and 3) each of the random

effect parameters and random errors are independent. The linear mixed-model framework for

continuous Gaussian outcomes is well studied and was developed by Laird and Ware.1–9 Performance 20

in Organic Chemistry II is modeled as a linear combination of the student-level covariates and the

random error representing the influence of class i on the student that is not captured by the observed

covariates; the random cluster errors are added to the regression model to account for the correlation

2/4/21 Page 2 of 6

of the students within each class. The linear mixed effects models and corresponding standardized

effect sizes are provided below. 25

EXAMPLE CALCULATIONS OF PREDICTED EXAM SCORES To compare the performance of repeaters and non-repeaters who took OCI in the traditional format

and had the same demographic and previous academic performance, we can use the linear mixed

models to calculate the estimated differences in predicted exam scores. Using the final exam score

model presented in the manuscript for Years 3-5 (Table 7), the difference in the predicted exam 30

performance for Non-Repeaters and Repeaters is 27 points for those students who took OCI in the

traditional format.

Difference in Predicted Final Exam Scores for Non-Repeater and Repeaters who took the

Traditional Format for OCI = 35

(Predicted Score for the Non-Repeaters) - (Predicted Score for the Repeaters)

= (64.31 -11.67*0 - 27.32*0 - 7.51*FG + 3.82*PEER - 3.19*Female -0.26*SATMathstd +

0.77*SATWritingstd + 14.52*0*0 + 13.82*0*FG)

- 64.31 - 11.67*0 - 27.32*1 - 7.51*FG + 3.82*PEER - 3.19*Female -0.26*SATMathstd +

0.77*SATWritingstd + 14.7018*0*1 + 13.82*0*FG) 40

= (-27.32*0) - (-27.32*1)

= 27.32 points

Similarly, the difference in predicted exam scores between students in OCI (flipped) who are

equivalent in every way except for repeater status can be calculated using the linear mixed model. The

predicted difference between Non-Repeaters and Repeaters in terms of final exam performance for 45

students who took OCI in the flipped format is 13 points.

Difference in Predicted Final Exam Scores for Non-Repeater and Repeaters who took the

Traditional Format for OCI =

(Predicted Score for the Non-Repeaters) - (Predicted Score for the Repeaters)

2/4/21 Page 3 of 6

= (64.31 - 11.67*1 - 27.32*0 - 7.51*FG + 3.82*PEER - 3.19*Female -0.26*SATMathstd + 50


- 64.31 - 11.67*1 - 27.32*1 - 7.51*FG + 3.82*PEER - 3.19*Female -0.26*SATMathstd +


= (64.31 - 11.67*1 - 27.32*0 + 14.52*1*0) - (64.31 - 11.67*1 - 27.32*1 + 14.52*1*1)

= 12.80 points 55

LINEAR MIXED EFFECTS MODELS FOR ORGANIC CHEMISTRY II FINAL EXAM The main findings are stable when the model for the Organic Chemistry II final exam for years 3-5

includes only significant covariates.

Table S1. OC II Final Exam Linear Mixed Effects Model Including Only Significant Covariates

Coefficient Standard

Error Degrees of Freedom

Test Statistic p-value

(Intercept) 59.88 3.71 255 16.16 <0.001

Flipped -3.70 4.75 1 -0.78 0.579

Repeater -28.49 3.96 255 -7.20 <0.001

Flipped and Repeater 13.74 4.91 255 2.80 0.0055

R2 = .23

60

The model for the Organic Chemistry II final exam including all years shows a similar result as the

model for years 3-5.

Table S2. OC II Final Exam Linear Mixed Effects Model for Years 1-5


Error Degrees of Freedom Test Statistic p-value

Intercept 60.62 2.58 309 23.49 <0.001

2/4/21 Page 4 of 6

Flipped -6.47 3.66 3 -1.77 0.175

Repeater -20.33 2.91 309 -6.99 <0.001

First Generation -2.16 3.08 309 -0.70 0.483

PEER 1.77 2.27 309 0.78 0.436

Female -4.64 2.11 309 -2.19 0.029

Standardized SAT Math 0.41 1.22 309 0.33 0.741

Standardized SAT Writing 1.35 1.16 309 1.16 0.248


Flipped and First Generation 8.70 4.20 309 2.07 0.039

R2 = 0.25

LINEAR MIXED EFFECTS MODELS FOR ORGANIC CHEMISTRY II MIDTERM EXAMS 65 Results of the linear mixed effects models for midterm exams are similar to those found for the

final exams. A gap exists between Repeater and Non-repeater students in Organic Chemistry II exam

scores. This gap is narrowed for Repeater students who enrolled in the flipped version of Organic

Chemistry I.

Table S3. OC II Midterm Exam 2 Linear Mixed Effects Model for Years 3-5 70



Intercept 59.77 3.21 230 18.65 <0.001

Flipped -6.27 3.70 1 -1.70 0.339

Repeater -18.07 3.95 230 -4.58 <0.001

First Generation 1.44 4.32 230 0.33 0.739

PEER 0.18 2.70 230 0.07 0.946

Female -2.67 2.48 230 -1.08 0.282


2/4/21 Page 5 of 6



Flipped and First Generation -1.39 5.16 230 -0.27 0.788

R2 = 0.15

Table S4. OC II Midterm Exam 1 Linear Mixed Effects Model for Years 3-5



Intercept 64.80 5.59 231 11.60 <0.001

Flipped -12.15 6.73 1 -1.81 0.322

Repeater -19.17 3.83 231 -5.00 <0.001


PEER 2.74 2.66 231 1.03 0.304

Female -2.44 2.43 231 -1.01 0.316





R2 = 0.18

REFERENCES (1) Nelder, J. A.; Wedderburn, R. W. M. Generalized Linear Models. J. R. Stat. Soc. Ser. A 1972, 135

(3), 370–384. 75

(2) Liang, K.-Y.; Zeger, S. L. Longitudinal Data Analysis Using Generalized Linear Models. Biometrika

1986, 73 (1), 13–22.

(3) McCullagh, P.; Nelder, J. A. Generalized Linear Models (Chapman & Hall/CRC Monographs on

Statistics and Applied Probability), 2nd edition.; Chapman and Hall/CRC: London, 1989.

2/4/21 Page 6 of 6

(4) Goldstein, H. Multilevel Statistical Models, 2nd ed.; Halstead Press: New York, 1995. 80

(5) Snijders, T. A. B.; Bosker, R. J. Multilevel Analysis: An Introduction to Basic and Advanced

Multilevel Modeling; Sage Publications: London, 1999.

(6) McCulloch, C. E.; Searle, S. R. Generalized, Linear, and Mixed Models; Wiley: Hoboken, N.J.,

2001.

(7) Diggle, P.; Heagerty, P.; Liang, K.-Y.; Zeger, S. Analysis of Longitudinal Data, 2nd edition.; Oxford 85

Statistical Science Series; Oxford University Press, Usa: New York, 2002.

(8) Raudenbush, S. W.; Bryk, A. S. Hierarchical Linear Models: Applications and Data Analysis

Methods (Advanced Quantitative Techniques in the Social Sciences), 2nd edition.; SAGE

Publications, Inc: Thousand Oaks, CA, 2001.

(9) Laird, N. M.; Ware, J. H. Random-Effects Models for Longitudinal Data. Biometrics 1982, 38 (4), 90

963–974.

download fileview on ChemRxivCHEMRXIV-REV Getting Students Back on Track Statis... (178.53 KiB)



2/4/21 Page 1 of 29

Getting Students Back on Track: Persistent Effects of Flipping Accelerated Organic Chemistry on Student Achievement, Study Strategies, and Perceptions of Instruction Lynn C. Reimer1,†, Kameryn Denaro2, Wenliang He3,††, Renée D. Link*,4

1Education Programs, University of California—Merced, Merced, California, 95343, United States 5

2Division of Teaching Excellence and Innovation, University of California—Irvine, Irvine, California,

92697, United States

3School of Education, University of California—Irvine, Irvine, California, 92697, United States

4Department of Chemistry, University of California—Irvine, Irvine, California, 92697, United States

ABSTRACT 10 Converting a first-term, accelerated summer organic chemistry course to a flipped format reduced the

achievement gap in the flipped course and in the second-term traditional lecture course between Non-

Repeaters taking an accelerated course to “get ahead” and Repeaters taking the course to “get back on

track.” The difference in final exam performance in the second-term course was nearly halved, the GPA

gap in both courses was reduced, and the gap in passing rate for the second-term course was 15

eliminated. First-generation students who took the first-term course in the flipped format experienced

a final exam score boost in the second-term course regardless of repeater status. While most students

responded positively to the flipped course structure, repeating students held a stronger preference for

the flipped format. These findings provide guidance on how to create courses that promote equity,

access and retention of diverse students in STEM. 20

2/4/21 Page 2 of 29

GRAPHICAL ABSTRACT

KEYWORDS Second-year Undergraduate, Organic Chemistry, Constructivism, Student-Centered Learning,

Learning Theories, Chemical Education Research 25

INTRODUCTION First-generation, low-income, PEERs (Persons Excluded from science because of Ethnicity or Race),

and women are entering science, technology, engineering, and math (STEM) majors, yet a

disproportionate number graduate in non-STEM disciplines.1–4 Only forty percent obtain a STEM

degree within six years.3 One of the most challenging introductory science courses is organic 30

chemistry, with failure and withdrawal (DFW) rates often higher than other lower division STEM

courses.5,6 Traditional lectures continue to dominate at larger universities, in which students passively

take notes and complete homework after class, with minimal opportunity for immediate feedback, peer

interaction, or collaborative problem solving.7 This style of instruction promotes independent learning

and thus is a cultural mismatch for certain students in STEM who rely on interaction with their peers 35

and are more successful when part of a community of learners (first-generation, low-income, PEERs

and women).8–10 Thus, partial or complete flipped instruction has increased to address these

concerns.11 In general, flipped instruction refers to any course where content delivery is completed

before class through videos and textbooks, freeing up class time for student-centered, active

2/4/21 Page 3 of 29

learning.11 Frequently larger classes include an electronic response system and/or peer leaders to: 40

gauge completion of pre-class work; formatively assess students during learning activities; facilitate

peer-peer interaction; and increase engagement.12–14

FLIPPED INSTRUCTION Numerous studies on flipped instruction in undergraduate STEM courses show positive results.

These studies typically focus on outcomes in the current course, rather than long-term benefits. Less 45

is known about the effects of flipped instruction in large enrollment, organic chemistry courses at

large, public research universities when taught in an accelerated summer term. This study

implemented a course redesign in an accelerated summer Organic Chemistry course that includes

ethnically and financially diverse students.

Flipped Instruction in Large General and Organic Chemistry Courses 50 Several studies of large, flipped general chemistry courses have reported positive outcomes,

including increased grades and decreased DFW rates (Table 1), affirming that flipped instruction may

be especially beneficial to certain subgroups of students.12,15–17 A subset of these studies identified

differing outcomes for different student groups. He, Holton, and colleagues found that only

sophomores and female students seemed to benefit from the flipped instruction, as measured by both 55

final exam outcomes, and post-course performance (even though there were numerous technology

challenges).18 Comparing flipped and traditional general chemistry courses, Ryan and Reid found a

small increase in final exam scores for students performing in the lower third of the class based on

pretest scoring.19 Deri et al. demonstrated a decrease in DFW rates and an increase in course grades

compared with historical trends of traditional lecture general chemistry courses enrolling up to 1,000 60

students per class across two campuses of a large, urban, public university.20 Importantly, this work

indicated larger gains for first-generation college students with lower SAT scores and coming from

lower-performing high schools. In a recent study of a large general chemistry course, Bancroft et. al,

found that a flipped format closed the performance gap between Black and Latinx students and their

White and Asian counterparts.21 The few studies on flipped learning in organic chemistry affirm an 65

advantage for students that might not otherwise succeed (Table 2).22–25 Crimmins and Midkiff found

students in a flipped course scored higher on the final exam and earned overall higher course grades

2/4/21 Page 4 of 29

compared with the historical traditional course; students in the 25th and 50th percentile experienced

the greatest benefit.25

Table 1. Findings of Studies on Flipped Large General Chemistry Courses 70

Study Authors Course Structure

Course Enrollment

Findings

Yestrebsky15 fully flipped 415 Small increase in course grades

Eichler and Peeples12

partially flipped 452 Increase in course grades, decrease in DFW rate

Bokosmaty, et al.16 partially flippeda 208-867 Increase in course grades, decrease in DFW rate

He, et al.17 fully flipped 334 Increase in final exam scores, no increase in study time

He, Holton, and colleagues18

fully flipped with technology challenges

313 Increase in final exam scores and post-course performance for sophomore and female students

Ryan and Reid19 fully flipped 117-206 Small increase in final exam scores for students in lower third of class

Deri, et al20 fully flipped 20-1000 Increase in course grades, decrease in DFW rate, larger gains for potentially less well-prepared entering college

Bancroft, et al.21 fully flipped 124 Performance gap closed between Black and Latinx students and White and Asian students

aIncludes introductory and general chemistry courses.

Table 2. Findings of Studies on Flipped Large Organic Chemistry Courses

Study Authors Course Structure Course Enrollment

Findings

Flynn22 fully flipped 17-400 Small increase in final exam scores, decrease in DFW rate.

Rein and Brookes23 partially flipped 192-222 No change in exam scores or DFW rate. Small positive change in course evaluations

Mooring, et al.24 fully flipped 212 No change in exam scores, increase in A and B grades, decrease in DFW rate, positive effect on emotional satisfaction and intellectual accessibility

2/4/21 Page 5 of 29

Crimmins and Midkiff25

fully flipped 395 Increase in final exam scores and course grades, greater benefit to students in 25th and 50th percentiles

Persistent Effects of Flipped Instruction Studies on longer term effects of flipped instruction are minimal (Table 3). In a study by Hibbard, a 75

small but statistically significant increase in student performance on the cumulative ACS standardized

exam was observed after a full year of a semi-self-paced flipped general chemistry course format.26 He,

Holton, and Farkas found that students in a partially flipped general chemistry course showed an

overall positive effect on end of course motivation and post-course grades, approximately half a letter

grade higher on average.27 A differentiated effect was seen, whereby students for whom traditional 80

instruction was not successful, showed higher motivation increases and higher gains in post-course

motivation after participating in flipped learning. Eichler and Peeples reported a GPA increase in

organic chemistry courses for the least academically prepared third of a cohort of students after a full

year of flipped general chemistry courses.28

Table 3. Persistent Effects from Flipped Chemistry Courses 85

Study Authors Course (Type)

Course Enrollment

Findings

Hibbard, et al.26 General Chemistry (semi-self-paced, flipped, full years sequence)

20-37 Increase in student performance on cumulative ACS standardized exam

He, Holton, and Farkas27

General Chemistry (partially flipped)

223 Half letter grade increase in post-course grades. Higher gains for certain subgroups of students

Eichler and Peeples28

General Chemistry (partially flipped, year-long sequence)

250-300 GPA increase in organic chemistry sequence for students with initially lower GPAs

Studies of flipped courses have increased in recent years, but studies describing large organic

chemistry courses in a flipped format with diverse student populations are less common. Furthermore,

no studies to date have examined the long-term effect of a flipped organic chemistry course taught at

an accelerated pace, with a focus on students repeating the course after an unsuccessful attempt. The 90

current study implemented flipped instruction in an organic chemistry course as a means to increase

2/4/21 Page 6 of 29

long term student achievement and is unique in that it focused on an accelerated summer course with

a diverse student enrollment, including many students who had previously failed the course and

enrolled in the summer course to “get back on track.”8

THEORETICAL FRAMEWORK 95 Two bodies of research on student performance and persistence framed this work: culturally

responsive teaching and social/academic integration. It is well known that undergraduate STEM

courses at large research universities do not have equitable success rates; certain groups of students,

first-generation, low-income, PEERs (Persons Excluded from science because of Ethnicity or Race) and

women, experience higher DFW rates.3 One explanation is cultural mismatch — certain groups of 100

students are disadvantaged because of conflicts between implicit expectations for autonomy in

American higher education and cultural identity.10,29–31 Culturally responsive teaching creates a

learning environment that accommodates students’ interdependent learning norms, tapping into

students’ prior experiences and knowledge to increase student performance and persistence.32,33 For

example, in a culturally responsive classroom, the instructor serves as facilitator. For the flipped 105

course in this study, the instructor provided in-class, collaborative activities including working

problems on paper, using model kits to investigate three-dimensional structures, and acting out

chemical processes. This embodies culturally mediated instruction by facilitating interdependent

learning norms that rely on peers helping one another. As a result, students are empowered, in control

of their learning, and able to function as a community, matching their cultural norms. By providing a 110

culturally responsive learning environment for students, they are able to integrate socially and

academically, which is known to contribute to persistence.34 Tinto’s model of social and academic

integration assumes students who persist in college and graduate participate in school culture.35

Unfortunately, strategies for increasing social integration are frequently extra-curricular, co-

curricular, or supplemental to classroom instruction. This study sought to increase social and 115

academic integration within the classroom by employing culturally responsive teaching, thereby

improving student performance, persistence, and matriculation into STEM careers.31,36 The flipped

model employed in this study creates the sense of belonging that previously was only available outside

of the classroom, and all students have access to integrate into the college culture. Flipped instruction

2/4/21 Page 7 of 29

combines the informal social integration with the formal academic integration to take advantage of the 120

one time all students are guaranteed to benefit. For students repeating a course, who are trying to “get

back on track” during an accelerated summer term, using culturally responsive teaching to promote

social and academic integration may help reduce the achievement gap in performance and persistence.

RESEARCH QUESTIONS To determine whether a flipped format in an accelerated summer organic chemistry class is 125

associated with a positive effect on student performance and persistence, we sought to answer the

following questions:

• How do students’ academic performance in an accelerated Organic Chemistry II (OCII) course

compare between students who took the flipped, accelerated Organic Chemistry I (OCI) course

and students who took the traditional, accelerated OCI course?37 130

• Will a flipped, accelerated OCI format reduce the achievement gap in OCI and OCII for students

who have unsuccessfully attempted OCI during a regular academic term?

• What were students’ perceptions of the course structure and their own skill development in the

flipped OCI course?

METHODS 135

Participants and Setting This study was approved by the Institutional Review Board as exempt (IRB 2012-8939 and IRB

2018-4211). The current study included five consecutive accelerated summer terms from 2009 to

2013, at a large public research university in the western United States with a diverse student

population.38 In the final year of the study, students self-reported as 43% Asian, 22% Hispanic/Latinx, 140

16% non-resident, 11% white, 4% other (Black, Pacific Islander, and two or more races), and 4% who

declined to state. Additionally, 49% of the students self-reported as first-generation college students

and 35% were identified as low-income based on self-reporting in admissions files. First-generation

status was defined as neither parent completing a 4-year degree (i.e. students with the highest level of

education being “some college”, “high school”, or “some high school”). Students who did not self-report 145

income were assumed to be non-low income students. The student population at the time of the study

was 50% male and 50% female based on self-reported responses to a binary choice question about

biological sex at time of admission.

2/4/21 Page 8 of 29

Each summer, Instructor 1 taught OCI in the five-week Summer Session I, while Instructor 2

taught OCII in the five-week Summer Session II term (Figure 1). In both accelerated courses, students 150

met for two-hour sessions three times a week. Instructor 1 used traditional face-to-face instruction

from Year 1–3 and implemented fully flipped instruction in all class meetings in Years 4 and 5.

Traditional lecture format for OCI included narrative lecture during which the instructor used a tablet

to fill in notes on slides and work example problems in class. Students were provided with the skeletal

notes in PDF format before class, and 3–5 clicker questions were given in each two-hour class meeting. 155

Students were encouraged to discuss responses with their peers. Instructor 2 used a similar

traditional lecture format in OCII during all years, including intermittent clicker questions. Students

purchased a bound notes packet that contained descriptive information, some pre-drawn structures,

and space for students to draw supplemental structures along with the instructor during the lecture.

During Years 3–5 (OC I: n3 = 219, n4 = 259, n5 = 210), enrollment in Organic Chemistry I courses was 160

consistently higher than in Year 1 and Year 2 (OC I: n1 = 171, n2 = 140), and met the standard criteria

for large enrollment.8

2/4/21 Page 9 of 29

Figure 1. Five-year Course and Final Exam Formats for Summer Organic Chemistry I and Organic Chemistry II 165

In the data analysis, Year 3 was chosen as the baseline for final exam consistency, as the final

exam administered in OCII was identical in Years 3–5 and differed slightly from the final exam

administered in Years 1 and 2. Midterm exam questions in OCII were isomorphic in nature, measuring

the same set of concepts with cosmetic changes in detail. To maintain study integrity, students had 170

access to graded midterm exams, but never graded final exams. Additionally, Year 3 was the third

consecutive summer that Instructor 1 had taught the OCI course in traditional format, theoretically

2/4/21 Page 10 of 29

providing a stable comparison between the traditional and flipped results, as the instructor would be

familiar with teaching this course in the accelerated context.

Students made enrollment decisions prior to the first summer session, making it highly unlikely 175

for students to make enrollment decisions for the second session based on their learning experience in

the first session. Because the courses were offered in accelerated summer terms, two distinct student

populations were present: a) “Non-Repeaters,” those seeking to accelerate their progress towards

degree completion, having not completed any prior organic chemistry courses; and b) “Repeaters,”

those students who had previously taken OCI and earned grades of C- or below. During the study, 979 180

students enrolled in OCI and 845 students enrolled in OCII; 378 students took both courses

consecutively, of whom 44% (168 students) were Repeaters taking OCI after an unsuccessful attempt

in the previous academic year. Of the 168 Repeater students enrolled in OCI, 11 were enrolled after

multiple failed attempts in the prior academic year (Table 4). Data from Years 1 and 2 is included for

reference purposes and to demonstrate any differences year over year of student enrollment in the 185

accelerated summer courses. This also allowed a larger sample size in our first analysis of student

demographics.

Table 4. Accelerated OCII Enrollment by OCI Status

Organic Chemistry I Summer Status

Non-Repeater Repeater Did not take consecutively

Total

Year 1 36 (26%) 22 (16%) 78 (57%) 136

Year 2 40 (29%) 11 (8%) 88 (63%) 139

Year 3 61 (38%) 35 (22%) 66 (40%) 162

Year 4 41 (24%) 53 (31%) 76 (45%) 170

Year 5 32 (13%) 43 (18%) 163 (68%) 238

Total 210 (25%) 168 (20%) 471 (55%) 845

2/4/21 Page 11 of 29

Final Study Population 190 To be included in this study, students agreed to the study conditions about data collection,

enrolled in OCI and OCII in Summer, and were fully matriculated students at the university. Students

enrolled in OCII in Summer but not exposed to the treatment during OCI (traditional vs flipped

classroom) were removed from the analysis. After meeting all study requirements, 250 students with

complete data, representing multiple forms of diversity, were included in the final modeling. The 195

sample consisted of 54% repeaters, 62% female, and 41% first-generation college students (Table 5).

Additionally, students who self-identified as Black, African American, Latino, Spanish American,

Chicano, Mexican American, American Indian, and Alaskan Native were classified as PEERs. The

sample average SAT Total score was 1770. Two policy changes enacted in Year 4 created a cost savings

for summer school, encouraging more low income students to enroll in summer courses. A policy to 200

allow the use of financial aid to pay for summer school and a “Pay for 8” policy where students could

take unlimited units for the same cost as eight units were introduced.

Table 5. Demographic Summary Statistics for OCII Students Who Took OCI in the Traditional Format (Year 3) and the Flipped Format (Years 4 and 5).

Traditional (Year 3) Flipped (Years 4-5)

Non- Repeater

Repeater Full Class

Non- Repeater

Repeater Full Classes Weighted Averagea

Female 46% 69% 54% 61% 73% 63%

PEER 30% 20% 26% 32% 15% 22%

First Generation

26% 31% 28% 32% 43% 38%

Low Income 13% 23% 17% 26% 29% 28%

SAT Math Mean (SD)

648 (74) 605 (85) 632 (80) 631 (79) 606 (100) 616 (93)

SAT Writing Mean (SD)

596 (88) 575 (81) 588 (86) 581 (99) 565 (86) 571 (92)

Sample Size 61 35 96 73 96 169

2/4/21 Page 12 of 29

aWeighted average accounts for the differing enrollments in Years 4 and 5.

205

Flipped Course Description: Organic Chemistry I Students were assigned material to review prior to class in a manner of their choosing — textbook

reading sections or short lecture videos. Copies of slides with blanks facilitated note-taking of the

material. Pre-class quizzes consisting of 3–5 multiple choice content questions and one open-ended

question regarding what the student found most difficult were used as an accountability mechanism 210

only to verify that students had completed the assigned reading or video segments rather than to

probe specific knowledge.

During class, students worked on problems designed to guide their progression through the

concepts presented in the pre-class materials. In-class activities included working problems on paper,

using model kits to investigate three-dimensional structures, and acting out chemical processes. For 215

example, during a class period focusing on conformational analysis, pairs of students built models

representing two similarly substituted cyclohexane structural drawings and worked together to

determine whether their models represented conformers or stereoisomers. When working on

nucleophilicity trends, a group of students volunteered to act out roles as solvent molecules and

nucleophiles of varying sizes, with the solvent molecule students attempting to “trap” the nucleophile 220

students. For all in-class activities, students were strongly encouraged to work with peers, but could

work alone if they preferred. Most students chose to work with classmates despite the difficulties for

facilitating group work presented by the fixed seating format of the classroom. Student work for in-

class activities was not collected and did not count for any credit toward final grades. Students were

incentivized to attend class because only solutions to in-class activities not completed during class 225

meetings were posted. Based on headcounts during randomly selected class meetings, approximately

70-85% of enrolled students attended.

After class, students completed homework through an online homework system for additional

practice. To keep the OCI workload hours comparable between the traditional and flipped courses, the

homework assignments for the flipped course were pared down versions of the assignments used in 230

the traditional lecture format. Any questions removed in the lighter homework assignments were

2/4/21 Page 13 of 29

moved to optional assignments that students could complete for practice but not for credit. In both

formats the course structure included a one-hour discussion section each week in which the graduate

student teaching assistant provided additional practice worksheets.

Measurements 235 Student exam performance, final course grades (GPA), and pass rate in OCI and OCII in Year 4 and

Year 5 were compared with Year 3 to identify any reduction in the achievement gap for students who

had previously failed OCI (Figure 1). Surveys were administered to complement the quantitative

analysis. Finding no appropriate pre-validated survey addressing perceptions of instructional

strategies and skill development, we developed a survey specifically for this study (Supporting 240

Information). In Year 4, we administered one survey at the end of the course. In Year 5 a modified

survey was administered at the beginning and end of the course. In developing the survey items, we

consulted with a colleague in Sociology involved in SoTL (Scholarship of Teaching and Learning) and a

colleague conducting similar qualitative research on flipped instruction.39–42 The open-ended questions

provided the richest data to identify any long-term benefits for students who have previously failed a 245

traditional organic chemistry course when retaking the course delivered in an accelerated, flipped

format.

Data Analysis Statistical analyses were performed to determine the effect of flipping a classroom during OCI on

college students’ performance in OCII. Performance in OCII was analyzed using linear mixed models in 250

three parts: score on the final exam; second midterm score; and first midterm score. Linear mixed-

effects models were fit to the data to account for the correlation of students nested within a class using

the open-source programming environment R as well as the lme4 and r2glmm packages.43–50 The

model was chosen to be parsimonious and included variables that have shown to be related to

performance in previous studies; covariates included student demographics (gender,9,51 first-255

generation college status,52 PEER status9), previous academic performance (OCI Repeater status,

standardized SAT Math scores, standardized SAT Writing scores53–55), and the type of OCI class

(traditional or flipped classroom). SAT Reading scores were not included in the model because of the

high collinearity with SAT Writing scores. To check for the collinearity of the remaining variables we

2/4/21 Page 14 of 29

calculated the correlation and the variance inflation factor (VIF) for the covariates in the model. The 260

correlation between the covariates were weak to moderate (r < |0.51|) and the VIF values were all

acceptable (VIF < 1.52). Both measures indicate that there are no issues with collinearity of our

covariates. A detailed description of the linear mixed effects models and the model results for

performance on the midterm exams are included in the Supporting Information.

To test the difference in the final exam score earned in OCII for students who took the OCI in the 265

flipped format versus those who took OCI in the traditional format, we used an independent samples t-

test. We also tested the difference in the pass rate for the two groups by testing the difference of the

two proportions. We hypothesized that the students who took the flipped format for OCI would score

higher in OCII and that they would have a higher pass rate in OCII compared to the students who took

OCI in the traditional format. 270

Survey items with Likert rankings were analyzed using a Wilcoxon rank sum test with continuity

correction (two-tailed) to identify any survey items that had statistically significant differences between

Non-Repeaters and Repeaters. Likert ratings were employed with a 5-point scale either “disagree-

agree” or “ineffective-effective,” as appropriate. We controlled the overall Type I error rate to be 0.05

using a Bonferroni correction (𝛼∗ = 𝛼/𝑚 = 0.05/12 = 0.004). 275

RESULTS Analysis of the quantitative and qualitative data show that students who previously attempted OCI

and took a flipped, accelerated format when repeating the course scored higher on exams during OCII,

earned higher grades in OCI and OCII, passed OCII on par with Non-Repeaters, and preferred the

flipped format.56 Table 6 shows summary statistics for Repeater and Non-repeater students and 280

provides evidence of the significant achievement gap in OCII between students who had

unsuccessfully attempted OCI previously and those who had not when taking an accelerated summer

course. The average final exam scores from summary statistics for Non-repeaters in OCII after OCI in

traditional format was 60 out of 100 points as compared to the Repeater’s average of 33 points. This

difference indicates a performance gap of 27 points between the two groups. 285

Final exam averages of Non-repeaters and Repeaters in OCII after OCI was taught in the flipped

format show a smaller performance gap of 14 points (Figure 2). Exam scores for Non-Repeaters

2/4/21 Page 15 of 29

appeared to decrease by four points, but this effect was not statistically significant (see below).

Repeaters scores, however, increased by 11 points. Midterm exams showed consistent trends

compared to the final exam. Although exam scores for all groups may appear low, an average exam 290

score of 60% is typical for organic chemistry courses at this university and does not represent a failing

grade. Final course letter grades are highly curved.

Table 6. Exam scores, Grades and Pass Rate Summary Statistics for OCII Students Who Took OCI in the Traditional Format (Year 3) and the Flipped Format (Years 4 and 5). Grades and exam scores are given as Mean (SD). 295

Traditional (Year 3) Flipped (Years 4-5)

Non- Repeater

Repeater Full Classes Weighted Average

Non- Repeater

Repeater Full Classes Weighted Average

OCII Midterm 1 62 (17) 41 (15) 54 (19) 56 (19) 49 (18) 52 (19)

OCII Midterm 2 57 (19) 39 (15) 50 (20) 53 (22) 45 (15) 49 (18)

OCII Final Exam 60 (20) 33 (14) 51 (22) 56 (20) 42 (18) 48 (20)

OCI Course GPA 2.80 (0.75) 2.01 (0.38) 2.51 (0.75) 2.93 (0.70) 2.83 (0.65) 2.87 (0.67)

OCII Course GPA 3.04 (0.97) 1.43 (1.10) 2.44 (1.28) 2.60 (1.23) 2.03 (1.00) 2.28 (1.14)

OCII Pass Rate 93% 60% 81% 82% 80% 81%

Sample Size 61 35 96 73 96 169

2/4/21 Page 16 of 29

Figure 2. Average exam scores on OCII final exams with standard errors, OCII course GPA with standard errors, and OCII pass rates indicate that Non-repeaters on average score better than Repeaters, and the gap between the two groups closes in OCII when OCI had flipped instruction. 300

The OCI course GPA, OCII course GPA, and the passing rate (final grade of C- or higher) for OCII

show a similar closing of the gap between Non-Repeater and Repeater students (Table 6). Students

repeating OCI in the traditional format earned lower final course grades than their Non-Repeater

counterparts in both OCI (0.79 GPA difference) and OCII (1.61 GPA difference). While 93% of Non-305

Repeaters passed OCII with a grade of C- or higher after taking OCI in traditional format, only 60% of

Repeaters passed OCII.

After OCI was converted to the flipped format, these gaps were decreased. Repeater students who

took flipped OCI had a final course GPA only 0.1 points below their Non-Repeater counterparts in OCI

and 0.57 points in OCII. The gap in the passing rate for OCII was eliminated as Repeaters passed OCII 310

at the same rate as Non-repeaters (Table 6, Figure 2). Although the final course GPA and the passing

rate for Non-Repeater students in OCII after flipped OCI did decrease, this change can be explained by

the class being graded on a competitive curve. The likelihood of a student earning a passing grade or

higher in OCII after the flipped OCI course was not dependent on whether or not they had repeated

OCI. A larger number of Repeater students scored above the passing threshold in the curved OCII 315

class after taking OCI in flipped format while some Non-Repeaters did not, again emphasizing that the

gap between these two groups was eliminated. The linear mixed effects model for OCII final exam

2/4/21 Page 17 of 29

scores is given in Table 7. The standardized effect size (R2) for the model was found to be 0.28,

indicating the 28% of the variation in OCII final exam scores can be explained by the combination of

the course format, student demographics, and whether or not the student was repeating the course. 320

Table 7. OCII Final Exam Linear Mixed Effects Model for Years 3-5

Coefficient Standard Error

Degrees of Freedom

Test Statistic p-value

(Intercept) 64.31 5.39 223 11.93 <0.001

Flipped -11.67 6.47 1 -1.80 0.323

Repeater -27.32 4.08 223 -6.70 <0.001


PEER 3.82 2.81 223 1.35 0.177

Female -3.19 2.55 223 -1.25 0.212





R2 = 0.28

Taking OCI in flipped format for Non-Repeaters did not have a statistically significant effect on

OCII final exam performance (coefficient = - 11.67, p = 0.323). Similarly, effects of demographic

variables and SAT scores were not significant. The achievement gap for Repeaters and Non-Repeaters 325

was smaller in the flipped format compared to the traditional format. While being a Repeater in OCI

resulted in a lower OCII estimated final exam score (coefficient = -27.32, p <0.001), Repeaters who

took OCI in the flipped format experienced an estimated OCII final exam score boost (coefficient =

14.52, p = 0.004). Non-Repeater students and Repeater students with similar demographics and SAT

2/4/21 Page 18 of 29

scores who took OCI in traditional format would have an estimated final exam score gap of 27.32 330

points, whereas equivalent Non-Repeater and Repeater students who took OCI in the flipped format

would have an estimated final exam score gap of only 12.80 points (see Supporting Information for

calculation).

In tests for two-way interactions between the course format and demographic characteristics, a

significant interaction was found only for first-generation status and course design (Table 7). First-335

generation students who took OCI in the flipped format experienced an estimated OCII final exam

score boost (coefficient = 13.82, p = 0.010). Tests for the three-way interaction between course design,

repeater status and first-generation status showed no significant interaction, indicating that first-

generation students benefited from the OCI flipped course format regardless of repeater status. While

Repeater first-generation students experienced the same closing of the exam performance gap with 340

their Non-Repeater counterparts (27.32 points in OCII after OCI in traditional format versus 12.80

points in OCII after OCI in flipped format), Non-Repeater first-generation students outperformed Non-

repeater students who did not self-identify as first generation. These Non-Repeater students and Non-

Repeater first-generation students with similar demographics and SAT scores who took OCI in

traditional format would have an estimated final exam score gap of 7.52 points. The model indicated 345

that the predicted exam score gap between equivalent Non-Repeater students (who do not identify as

first generation) and Non-Repeater first-generation students who took OCI in the traditional format

(7.52 points) was completely closed when the students took OCI in the flipped format. In fact,

predicted exam scores for Non-Repeater first generation students were 6.31 points higher than their

counterparts who did not identify as first generation. 350

Results from the surveys support the quantitative findings.57–61 The open-ended questions provide

qualitative evidence that the flipped instruction in OCI was helpful and changed students’ study

habits. Table 8 includes representative student quotes. Students’ own wording was used to identify

themes based on identified relations, similarities and differences that were grouped conceptually.62

Common themes described the learning experience as: demanding, engaging, lots of problem solving 355

(with peers), interactive, helpful demonstrations, hands-on (molecular model kits) and gaining a

2/4/21 Page 19 of 29

deeper understanding. Common caveats included: more work (for students and instructors), more

responsibility, and no room to procrastinate.

Table 8. Flipped Organic Chemistry I Student Post-Survey Representative Examples of Common Responses (Year 4 and Year 5) 360

What did you think of the “inverted” method of teaching the course?

What in-class methods did you find especially helpful?

What out-of-class methods did you find especially helpful?

How did your study habits change in the “inverted” class?

Demanding; Deeper understanding; Engaging; Interactive;

Solving problems; Building with model kits; iClickers; Demonstrations; Peer discussions

Podcasts; Quizzes before class; Self-study strategies; Online homework; Discussion section

Study daily; No room to procrastinate; More ownership

I failed my first OChem course...this method has helped me learn and succeed...allowed me to fully understand the fundamentals...who would have thought I would have a strong interest in organic chemistry

Working on problems in which you are given time to work with peers and listen to their thoughts and opinions on the subject, seeing how they personally work on problems...may see another approach towards a problem.

Encouraging the podcasts and readings BEFORE class...Requiring Sapling homework after each podcast...forced students not to procrastinate… actually helped learn the material instead of cram before exams.

I am more active in studying and less hesitant to attempt problems...because I’ve learned the common mistakes in class and have my professor...for questions or struggles I may be facing.

It was a little harder since we, as the students, definitely had a lot more responsibility since we had to take the time to review the subjects we would be talking about in lecture...hard to follow along when I skipped a podcast or two because of lack of time management on my part...

I thought working on problems during class slowly was helpful in the chapters that were particularly difficult… Using the 3D model kit was VERY helpful for me and I appreciated how the TA and [Instructor] would walk around to check if students had the right structure.

I felt the podcasts were the most helpful...gave me the ability to listen to the lecture at my own speed...pause...when I didn’t fully understand… allowed me to be more productive in class by actually applying the concepts...

With this method, there is no room nor any reason to procrastinate so I just did it, I just studied easily and smoothly...never felt rushed to move onto the next topic...knew there would be time to ask questions and practice with my peers and instructor...I have a strong foundation in O-chem...feels awesome...

I thought it was useful and more efficient since we were able to do hands on problems during lecture. However for the students that didn’t watch the podcasts, lecture was pretty much useless since they didn’t know what is going on. It makes you better prepared and avoids procrastination.

I especially liked doing practice problems with the class as a whole...awesome when [Instructor] and TA were going around the huge lecture hall answering questions...made me feel like it was a small, interactive class...liked that we could actually keep up with the pace and learn every step.

Podcasts, since I could review them if needed, and the online pre-class quizzes, so I could see how well I really understood the material and decide what I need to focus my attention on when studying.

It made me realize how much better it is to be prepared… online quizzes...forced me to watch the podcasts and learn the material before class...I always knew what was going on and was never confused… study habits have slowly involved much less procrastinating… grew to appreciate a bit of studying each day

2/4/21 Page 20 of 29

Finally, the Likert scale items support the qualitative, open-ended results (Table 9). The majority of

students felt that the flipped format was effective and would prefer to take more flipped courses. The

findings are even more compelling when Non-Repeaters and Repeaters are separated. We used

Wilcoxon rank sum tests with continuity corrections (two-tailed) to identify any items that had 365

statistically significant differences between Non-Repeaters and Repeaters, and we controlled the

overall Type I error rate to be 0.05 using a Bonferroni correction (𝛼∗ = 𝛼/𝑚 = 0.05/12 = 0.004).

Repeaters’ preferences for the flipped format were statistically higher than Non-Repeaters (Table 9).

Because many changes were made to the survey in Year 5, we include it in the Supporting Information

for reference. 370

Table 9. Comparison of Non-Repeaters and Repeaters on Year 4 Survey Items

Survey Questiona Non-Repeaters mean (sd)

Repeaters mean (sd)

p-value

I would recommend an inverted class to my friends who need to take organic chemistry.

3.72 (1.19) 4.26 (0.91) 0.001

I think listening to lecture outside of class and working on problem solving in class is an effective way to learn.

3.77 (1.19) 4.42 (0.84) <0.001

I would prefer to take more science classes that use this type of class format.

3.69 (1.23) 4.21 (0.91) 0.004

aLikert rankings were either “disagree-agree” or “ineffective-effective,” as appropriate to the survey statements and employed a 5-point scale

DISCUSSION The change in OCI course structure reduced the achievement gap between Non-Repeaters taking

an accelerated course to “get ahead” and Repeaters taking the course to “get back on track”. The 375

difference in final exam performance was nearly halved, the GPA gap was reduced, and the gap in

passing rate was eliminated. The decrease in Non-Repeaters’ outcomes between the flipped and

traditional course formats is not statistically different (Table 7). Additionally, first-generation students

experienced a benefit in OCII from the flipped OCI course format regardless of repeater status; Non-

Repeater first-generation students outperformed those Non-Repeaters that did not self-identify as first-380

generation. Because the courses were offered in the accelerated summer term, one would expect a

2/4/21 Page 21 of 29

significant achievement gap between the “Non-Repeaters,” those seeking to accelerate their progress

towards degree completion, and the at-risk “Repeaters,” who had previously failed OCI one or more

times. However, converting OCI to a flipped format eliminated the gap in passing rate for Repeaters in

OCII, potentially increasing Repeaters’ chances of graduating in STEM and pursuing their career 385

goals. The increased performance for first-generation students, regardless of repeater status, is

consistent with previous studies on cultural mismatch.10 When the learning environment matches

first-generation students’ interdependent learning norms, they perform academically on par with

students accustomed to an independent learning norm, more typical of American Universities.10

Studies that show a gain within the flipped course may be promising, but to show that the gain 390

persists throughout the second course is compelling.

The qualitative data further supports that there is a difference between Non-Repeaters and

Repeaters in the accelerated summer course. For certain survey items (Table 9) there was a

statistically positive difference for Repeaters in preference for a flipped format that guides how they

study for the course outside of class (i.e., watching lectures) to prepare for working on problems in 395

class, potentially increasing their performance in other STEM courses. Repeaters reported they would

like to take more STEM courses in a flipped format.34 These findings are consistent with previous

studies on flipped organic chemistry courses.24,25,41,63–65

Comparison of final exam scores, course grades and passing rate in OCII affirm that flipping the

accelerated OCI course — culturally responsive teaching — promotes equity beyond access to include 400

persistence for undergraduate STEM majors. The course format provided opportunities for Repeaters

to develop a stronger sense of social and academic integration as they learned problem solving skills

alongside their peers. Research shows this leads to improved academic performance.35 The flipped

format provided many avenues by which students’ academic potential could be reached. Random

headcounts revealed that approximately 70-85% of enrolled students attended class, where they were 405

strongly encouraged (but not required) to work with peers. Such a strong attendance rate affirms

students’ positive feedback regarding course structure. Students found the format to be engaging,

active, more personal, and hands-on; they valued peer discussions, demonstrations, building

molecular models with physical kits, covering the “tricky” practice problems, and clicker questions as

2/4/21 Page 22 of 29

a means to foster discussion. Students participated in activities intentionally designed to foster social 410

integration, guide their academic progression and increase their academic integration. For example,

pairs of students worked together to build models of structural drawings while solving problems. On

other occasions, students volunteered to act out roles, such as solvent molecules and nucleophiles of

varying sizes. These activities created an environment of interdependent learning whereby students

were able to work with one another and learn from one another towards a common goal.10 This flipped 415

course modeled culturally responsive teaching by matching the students’ diverse learning needs. In

culturally responsive teaching, we refer to these as multiple modalities. In the classroom, these

include: hands-on activities (inclusive of molecular model kits), peer problem solving, and embodied

learning. Outside of the classroom, these include: podcasts, reading, note taking, online homework

and online quizzes. Another aspect of culturally responsive teaching includes reteaching where content 420

is presented in new and different ways that attend to students’ needs.66 Providing multiple

instructional modalities in person and online increased the learning opportunities for all students. The

flipped course also taught and modeled study strategies explicitly, rather than assuming the students

were enculturated in the norms of higher education. Students specifically described how their study

habits changed. It is likely these new skills transferred to the next, traditionally taught course. The 425

flipped format not only provided a deeper understanding of the fundamental concepts of Organic

Chemistry I, it provided skill development, applicable to future courses, enhancing academic

integration.

LIMITATIONS While this study is potentially generalizable to other public research institutions, there are 430

noteworthy limitations. Although surveys were used to support quantitative results, further studies

are needed to establish the surveys’ validity. The institution’s selectivity may also have blunted the

statistical significance of findings. To enroll in organic chemistry, students must have successfully

completed the first-year chemistry series. Furthermore, in order to enroll in the first course of first-

year chemistry, students must have scored above 600 on SAT Math or completed an additional 435

chemistry preparatory course. Therefore, those students taking organic chemistry have had initial

success in their STEM major, but are at risk of dropping out of STEM during their second year.2,9

2/4/21 Page 23 of 29

FUTURE RESEARCH Additional studies of accelerated summer STEM courses employing culturally responsive teaching

are needed, including studies on effective reteaching for students who are retaking courses and 440

include a disproportionate number of students who self-report as first-generation, low-income, PEERs,

and women. This is especially true for developmental courses that are often taught using pre-packaged

online programs with fewer instructional modalities and taken independently by students to get

“caught up.” This course structure results in minimal opportunity for social or academic integration.

Further qualitative research is needed to follow-up on student perceptions in subsequent courses after 445

they have taken flipped courses, especially those flipped courses that are culturally responsive and

engender social and academic integration. This qualitative research needs to focus on large enrollment

courses to complement the current research on smaller classes.

CONCLUSION Organic chemistry has been described as one of the most difficult STEM gateway courses, forcing 450

students out of their STEM major more than any other course and contributing to the persistence

achievement gap.5,6,9 Increased performance between an intervention and control group in the current

course is encouraging but anticipated. Improved outcomes — higher final exam scores, higher course

grades, and consistent passing rates between Repeaters and Non-Repeaters, and the increase in exam

scores for first-generation students regardless of repeater status — in the next course have 455

implications for STEM graduation rates and careers for all students. The flipped format of instruction

in this study models culturally responsive teaching in ways rarely seen at the undergraduate level, as

compared with K–12 classes.67,68 Reteaching material to students (Repeaters) is most successful when

it is engaging and interactive (group work), information is in digestible chunks (video podcasts), and it

includes frequent formative assessment (clicker questions) and opportunities for practice.69,70 460

Furthermore, effective reteaching has not been studied as well at the undergraduate level because

students typically retake failed courses in the same traditional format as the original course. The

findings here affirm that while flipped instruction may be generally preferred and helpful to all

students, its significance lies in its commitment to culturally responsive teaching, and appropriately

2/4/21 Page 24 of 29

reteaching previously unsuccessful students, rather than weeding these students out of STEM 465

courses, majors and careers.

ASSOCIATED CONTENT Supporting Information

The Supporting Information is available:

Survey Instruments (PDF) 470

Statistical Models (PDF)

Survey Data Analysis (PDF)

AUTHOR INFORMATION Corresponding Author *E-mail: [email protected] 475

Notes †Current affiliation for Lynn C. Reimer: Madera Unified School District, Madera, CA 93637

††Current affiliation for Wenliang He: Zhejiang Normal University, Jinhua, Zhejiang, China

The authors declare no competing financial interests.

ACKNOWLEDGMENTS 480 The authors thank their respective universities for providing the support to conduct this study,

colleagues who graciously read versions of the manuscript, students in the Organic Chemistry

Summer Session courses for their participation, and Susan King for participating in the study. In

particular, we thank Amanda Holton for providing editorial comments. We thank Kelly Butzler for help

with help constructing the Year 4 survey and Jim Hull for help constructing the Year 5 survey. We 485

also wish to thank Kate McKnelly for the helpful suggestions that led to the creation of Figure 1.

Finally, we thank the RStats community for helpful R packages and trouble-shooting support.

REFERENCES (1) Students who self-identified as Black, African American, Latino, Spanish American, Chicano,

Mexican American, American Indian, and Alaskan Native were classified as PEERs. 490

(2) Chen, X. STEM Attrition: College Students’ Paths Into and Out of STEM Fields; National Center for

Education Statistics, Institute of Education Sciences, Ed.; U.S. Department of Education:

Washington, D.C., 2013.

2/4/21 Page 25 of 29

(3) Eagan, K.; Hurtado, S.; Figueroa, T.; Hughes, B. Examining STEM Pathways among Students

Who Begin College at Four-Year Institutions. National Academy of Sciences 2014. 495

(4) Tibbetts, Y.; Harackiewicz, J. M.; Priniski, S. J.; Canning, E. A. Broadening Participation in the

Life Sciences with Social-Psychological Interventions. CBE Life Sci. Educ. 2016, 15 (3).

https://doi.org/10.1187/cbe.16-01-0001.

(5) Grove, N. P.; Hershberger, J. W.; Bretz, S. L. Impact of a Spiral Organic Curriculum on Student

Attrition and Learning. Chem. Educ. Res. Pract. 2008, 9 (2), 157–162. 500

(6) Grove, N. P.; Bretz, S. L. Perry’s Scheme of Intellectual and Epistemological Development as a

Framework for Describing Student Difficulties in Learning Organic Chemistry. Chem. Educ. Res.

Pract. 2010, 11 (3), 207–211.

(7) Smith, M. K.; Vinson, E. L.; Smith, J. A.; Lewin, J. D.; Stetzer, M. R. A Campus-Wide Study of

STEM Courses: New Perspectives on Teaching Practices and Perceptions. CBE Life Sci. Educ. 505

2014, 13 (4), 624–635.

(8) Freeman, S.; Eddy, S. L.; McDonough, M.; Smith, M. K.; Okoroafor, N.; Jordt, H.; Wenderoth, M.

P. Active Learning Increases Student Performance in Science, Engineering, and Mathematics.

Proc. Natl. Acad. Sci. U. S. A. 2014, 111 (23), 8410–8415.

(9) Harris, R. B.; Mack, M. R.; Bryant, J.; Theobald, E. J.; Freeman, S. Reducing Achievement Gaps 510

in Undergraduate General Chemistry Could Lift Underrepresented Students into a

“hyperpersistent Zone.” Science Advances 2020, 6 (24), eaaz5687.

(10) Stephens, N. M.; Fryberg, S. A.; Markus, H. R.; Johnson, C. S.; Covarrubias, R. Unseen

Disadvantage: How American Universities’ Focus on Independence Undermines the Academic

Performance of First-Generation College Students. J. Pers. Soc. Psychol. 2012, 102 (6), 1178. 515

(11) Seery, M. K. Flipped Learning in Higher Education Chemistry: Emerging Trends and Potential

Directions. Chem. Educ. Res. Pract. 2015, 16 (4), 758–768.

(12) Eichler, J. F.; Peeples, J. Flipped Classroom Modules for Large Enrollment General Chemistry

Courses: A Low Barrier Approach to Increase Active Learning and Improve Student Grades.

Chem. Educ. Res. Pract. 2016, 17 (1), 197–208. 520

(13) Reimer, L. C.; Schenke, K.; Nguyen, T.; O’dowd, D. K.; Domina, T.; Warschauer, M. Evaluating

Promising Practices in Undergraduate STEM Lecture Courses. RSF: The Russell Sage Foundation

Journal of the Social Sciences 2016, 2 (1), 212–233.

(14) Talbot, R. M.; Hartley, L. M.; Marzetta, K.; Wee, B. S. Transforming Undergraduate Science

Education With Learning Assistants: Student Satisfaction in Large-Enrollment Courses. J. Coll. 525

Sci. Teach. 2015, 44 (5), 24–30.

(15) Yestrebsky, C. L. Flipping the Classroom in a Large Chemistry Class-Research University

Environment. Procedia Soc Behav Sci 2015, 191, 1113–1118.

(16) Bokosmaty, R.; Bridgeman, A.; Muir, M. Using a Partially Flipped Learning Model To Teach First

Year Undergraduate Chemistry. J. Chem. Educ. 2019, 96 (4), 629–639. 530

2/4/21 Page 26 of 29

(17) He, W.; Holton, A.; Farkas, G.; Warschauer, M. The Effects of Flipped Instruction on out-of-Class

Study Time, Exam Performance, and Student Perceptions. Learn Instr 2016, 45, 61–71.

(18) He, W., Holton, A.J., Farkas, G. Differentiated Impact of Flipped Instruction: When Would

Flipped Instruction Work or Falter? Learn. Teach. 2019, 1 (31).

(19) Ryan, M. D.; Reid, S. A. Impact of the Flipped Classroom on Student Performance and Retention: 535

A Parallel Controlled Study in General Chemistry. J. Chem. Educ. 2016, 93 (1), 13–23.

(20) Deri, M. A.; Mills, P.; Mc Gregor, D. Structure and Evaluation of a Flipped General Chemistry

Course as a Model for Small and Large Gateway Science. J Coll Sci Teach 2018, 47, 68–77.

(21) Bancroft, S. F.; Fowler, S. R.; Jalaeian, M.; Patterson, K. Leveling the Field: Flipped Instruction

as a Tool for Promoting Equity in General Chemistry. J. Chem. Educ. 2020, 97 (1), 36–47. 540

(22) Flynn, A. B. Structure and Evaluation of Flipped Chemistry Courses: Organic & Spectroscopy,

Large and Small, First to Third Year, English and French. Chem. Educ. Res. Pract. 2015, 16 (2),

198–211.

(23) Rein, K. S.; Brookes, D. T. Student Response to a Partial Inversion of an Organic Chemistry

Course for Non-Chemistry Majors. J. Chem. Educ. 2015, 92 (5), 797–802. 545

(24) Mooring, S. R.; Mitchell, C. E.; Burrows, N. L. Evaluation of a Flipped, Large-Enrollment Organic

Chemistry Course on Student Attitude and Achievement. J. Chem. Educ. 2016, 93 (12), 1972–

1983.

(25) Crimmins, M. T.; Midkiff, B. High Structure Active Learning Pedagogy for the Teaching of Organic

Chemistry: Assessing the Impact on Academic Outcomes. J. Chem. Educ. 2017, 94 (4), 429–438. 550

(26) Hibbard, L.; Sung, S.; Wells, B. Examining the Effectiveness of a Semi-Self-Paced Flipped

Learning Format in a College General Chemistry Sequence. J. Chem. Educ. 2016, 93 (1), 24–30.

(27) He, W.; Holton, A. J.; Farkas, G. Impact of Partially Flipped Instruction on Immediate and

Subsequent Course Performance in a Large Undergraduate Chemistry Course. Comput. Educ.

2018, 125, 120–131. 555

(28) Eichler, J. F.; Peeples, J. Flipped Classroom Learning Environments in General Chemistry: What

Is the Impact on Student Performance in Organic Chemistry? In Active Learning in General

Chemistry: Whole-Class Solutions; ACS Symposium Series; American Chemical Society, 2019;

Vol. 1322, pp 171–189.

(29) Inzlicht, M.; Schmader, T. Stereotype Threat: Theory, Process, and Application, 1st edition.; 560

Oxford University Press: Oxford, 2011.

(30) Steele, C. M. Whistling Vivaldi: How Stereotypes Affect Us and What We Can Do (Issues of Our

Time), Reprint edition.; W. W. Norton & Company: New York, 2011.

(31) Tinto, V. Completing College: Rethinking Institutional Action; University of Chicago Press: Chicago,

2012. 565

(32) Gay, G. Culturally Responsive Teaching: Theory, Research, and Practice (Multicultural Education

Series), 2nd edition.; Banks, J. A., Ed.; Teachers College Press: New York, 2010.

2/4/21 Page 27 of 29

(33) Martell, C. C. Race and Histories: Examining Culturally Relevant Teaching in the U.S. History

Classroom. Theory & Research in Social Education 2013, 41 (1), 65–88.

(34) Tinto, V. Dropout from Higher Education: A Theoretical Synthesis of Recent Research. Rev. Educ. 570

Res. 1975, 45 (1), 89–125.

(35) Severiens, S. E.; Schmidt, H. G. Academic and Social Integration and Study Progress in Problem

Based Learning. Higher Education 2008, 58 (1), 59.

(36) Engle, J.; Tinto, V. Moving Beyond Access: College Success for Low-Income. First-Generation

Students, Washington, DC: Pell Institute for the Study of Opportunity in Higher Education 2008. 575

(37) Organic Chemistry II was taught in a traditional, accelerated summer format throughout the

study. Organic Chemistry III was only offered during the regular academic terms.

(38) Regular terms are ten weeks, while summer terms are five weeks.

(39) Huber, M. T.; Albers, C. M.; Bennett, C. D.; Bernstein, J. L.; Bauer-Dantoin, A.; Chick, N. L.;

Christensen, T. B.; Dewar, J. M.; Diaz, A. J.; Gurung, R. A. R.; Schwartz, B. M.; Grauerholz, L.; 580

Gresalfi, M.; Kearns, K. D.; Main, E.; Mateiro, A. E.; Middendorf, J.; Pace, D.; Persell, C. H.;

Poole, G.; Randall, N.; Reichard, D. A.; Robinson, J. M.; Scharff, L.; Shopkow, L.; Sievert, A. K.;

Takayama, K.; Werder, C.; Zolan, M. E. The Scholarship of Teaching and Learning In and Across

the Disciplines; McKinney, K., Ed.; Indiana University Press: Bloomington, IN, 2013.

(40) Butzler, K. B. The Effects of Motivation on Achievement and Satisfaction in a Flipped Classroom 585

Learning Environment. Doctor of Education, Northcentral University, 2014.

(41) Fautch, J. M. The Flipped Classroom for Teaching Organic Chemistry in Small Classes: Is It

Effective? Chem. Educ. Res. Pract. 2015, 16 (1), 179–186.

(42) Fautch and the authors of this manuscript consulted with Dr. Kelly Butzler when designing

surveys, leading to very similar surveys used for each study. Dr. Butzler’s dissertation is also 590

cited.

(43) Theobald, R.; Freeman, S. Is It the Intervention or the Students? Using Linear Regression to

Control for Student Characteristics in Undergraduate STEM Education Research. CBE Life Sci.

Educ. 2014, 13 (1), 41–48.

(44) Theobald, E. Students Are Rarely Independent: When, Why, and How to Use Random Effects in 595

Discipline-Based Education Research. CBE Life Sci. Educ. 2018, 17 (3), rm2.

(45) R Core Team. R: A Language and Environment for Statistical Computing; R Foundation for

Statistical Computing: Vienna, Austria, 2019.

(46) Bates, D.; Mächler, M.; Bolker, B.; Walker, S. Fitting Linear Mixed-Effects Models Using lme4.

Journal of Statistical Software, Articles 2015, 67 (1), 1–48. 600

(47) Edwards, L. J.; Muller, K. E.; Wolfinger, R. D.; Qaqish, B. F.; Schabenberger, O. AnR2statistic for

Fixed Effects in the Linear Mixed Model. Statistics in Medicine. 2008, pp 6137–6157.

https://doi.org/10.1002/sim.3429.

2/4/21 Page 28 of 29

(48) Nakagawa, S.; Schielzeth, H. A General and Simple Method for Obtaining R2 from Generalized

Linear Mixed-Effects Models. Methods Ecol. Evol. 2013, 4 (2), 133–142. 605

(49) Jaeger, B. C.; Edwards, L. J.; Das, K.; Sen, P. K. An R2 Statistic for Fixed Effects in the

Generalized Linear Mixed Model. Journal of Applied Statistics. 2017, pp 1086–1105.

https://doi.org/10.1080/02664763.2016.1193725.

(50) Jaeger, B. r2glmm: Computes R Squared for Mixed (Multilevel) Models. R package version 0.1.2

https://cran.r-project.org/package=r2glmm (accessed Jan 20, 2020). 610

(51) Binary biological sex is being used as a proxy for gender because institutional data gathered at

the time of this study did not collect information on gender identity. Experiences of students

identifying as trans, nonbinary, or other genders are not reflected in this work due to this

limitation.

(52) Austin, A. C.; Ben-Daat, H.; Zhu, M.; Atkinson, R.; Barrows, N.; Gould, I. R. Measuring Student 615

Performance in General Organic Chemistry. Chemistry Education Research and Practice 2015, 16

(1), 168–178.

(53) Frey, R. F.; Cahill, M. J.; McDaniel, M. A. Students’ Concept-Building Approaches: A Novel

Predictor of Success in Chemistry Courses. J. Chem. Educ. 2017, 94 (9), 1185–1194.

(54) Lewis, S. E.; Lewis, J. E. Predicting at-Risk Students in General Chemistry: Comparing Formal 620

Thought to a General Achievement Measure. Chem. Educ. Res. Pract. 2007, 8 (1), 32–51.

(55) Pursell, D. P. Predicted versus Actual Performance in Undergraduate Organic Chemistry and

Implications for Student Advising. J. Chem. Educ. 2007, 84 (9), 1448.

(56) Students in Years 1 and 2 took the same final exam, but this exam was not identical to the exam

taken by students in Years 3-5. 625

(57) de la Rubia, E. A.; Zhu, H.; Ellis, S.; Quinn, M.; McNamara, A. skimr: Compact and Flexible

Summaries of Data https://CRAN.R-project.org/package=skimr (accessed Nov 12, 2020).

(58) Robinson, D.; Hayes, A. broom: Convert Statistical Analysis Objects into Tidy Tibbles

https://CRAN.R-project.org/package=broom (accessed Sep 12, 2019).

(59) Bryer, J. likert: Analysis and Visualization Likert Items http://github.com/jbryer/likert 630

(accessed Sep 22, 2019).

(60) Garnier, S. viridis: Default Color Maps from “matplotlib” https://CRAN.R-

project.org/package=viridis (accessed Nov 12, 2020).

(61) Wickham, H.; Averick, M.; Bryan, J.; Chang, W.; McGowan, L.; François, R.; Grolemund, G.;

Hayes, A.; Henry, L.; Hester, J.; Kuhn, M.; Pedersen, T.; Miller, E.; Bache, S.; Müller, K.; Ooms, 635

J.; Robinson, D.; Seidel, D.; Spinu, V.; Takahashi, K.; Vaughan, D.; Wilke, C.; Woo, K.; Yutani,

H. Welcome to the Tidyverse. JOSS 2019, 4 (43), 1686.

(62) Glaser, B.; Strauss, A. The Discovery of Grounded Theory. London: Weidenfeld and Nicholson

1967, 24 (25), 288–304.

2/4/21 Page 29 of 29

(63) Shattuck, J. C. A Parallel Controlled Study of the Effectiveness of a Partially Flipped Organic 640

Chemistry Course on Student Performance, Perceptions, and Course Completion. J. Chem. Educ.

2016, 93 (12), 1984–1992.

(64) Shattuck, J. C. Benefits of a Partially Flipped Organic Chemistry Course to Student Perceptions

and Learning. In Active Learning in Organic Chemistry: Implementation and Analysis; ACS

Symposium Series; American Chemical Society, 2019; Vol. 1336, pp 167–186. 645

(65) Tekane, R.; Pilcher, L. A.; Potgieter, M. Blended Learning in a Second Year Organic Chemistry

Class: Students’ Perceptions and Preferences of the Learning Support. Chem. Educ. Res. Pract.

2020, 21 (1), 24–36.

(66) Bellert, A. Effective Re-Teaching. Australian Journal of Learning Difficulties 2015, 20 (2), 163–

183. 650

(67) J. Mau, W.-C. Characteristics of US Students That Pursued a STEM Major and Factors That

Predicted Their Persistence in Degree Completion. ujer 2016, 4 (6), 1495–1500.

(68) Cohen, R.; Kelly, A. M. Mathematics as a Factor in Community College STEM Performance,

Persistence, and Degree Attainment. J Res Sci Teach 2020, 57 (2), 279–307.

(69) Marzano, R. J.; Pickering, D. J.; Pollock, J. E. Classroom Instruction That Works: Research-Based 655

Strategies for Increasing Student Achievement; Association for Supervision and Curriculum

Development: Alexandria, VA, 2001.

(70) Marzano, R. J. Reviving Reteaching. Educ. Leadersh. 2010, 68 (2), 82–83.

download fileview on ChemRxivCHEMRXIV - Getting Students Back on Track Rev 2.4.202... (1.14 MiB)



getting students back on track: persistent effects of

Documents