IDENTIFYINGPREDICTORSOFPERFORMANCEONUSMLE®STEP1
by
SACHINSHAH
DISSERTATION
PresentedtotheFacultyoftheMedicalSchoolTheUniversityofTexasSouthwesternMedicalCenter
InPartialFulfillmentoftheRequirementsFortheDegreeof
DOCTOROFMEDICINEWITHDISTINCTIONINMEDICALEDUCATION
TheUniversityofTexasSouthwesternMedicalCenterDallas,TX
©CopyrightbySachinShah2017AllRightsReserved
ACKNOWLEDGMENTS
ThisresearchwassupportedbytheUTSouthwesternOfficeofMedicalEducation,OfficeofStudentAffairs,andStudentAcademicSupportServices.Ithankmymentors,Dr.DorothySendelbach,Dr.AngelaMihalic,andDr.ArleneSachs,fortheirunwaveringsupportandguidancethroughoutthisresearchproject.IthankSholaRogersforhereffortscompilingthedatabase,andIthankMs.CarolWorthamandMs.AnneMcLanefortheirpartsin
compilingdata.IalsothankDr.JohnSadlerforhisguidanceinobtainingIRBexemption,andDr.JoanReischforherearlyinputonproposedstatisticalmethods.
1
TABLEOFCONTENTSINTRODUCTION....................................................................................................................................2BACKGROUND........................................................................................................................................3METHODS................................................................................................................................................7RESULTS................................................................................................................................................10DISCUSSION..........................................................................................................................................19CONCLUSION........................................................................................................................................25LISTOFTABLES...................................................................................................................................28LISTOFFIGURES.................................................................................................................................33REFERENCES........................................................................................................................................46VITAE......................................................................................................................................................48APPENDIX.............................................................................................................................................49
2
INTRODUCTION
USMLE®Step1isconsideredbyresidencyprogramdirectorstobeoneofthemost
importantfactorsinselectingmedicalstudentsforinterviews,soitisimportantforboth
studentsandmedicalschoolstomaximizescores1.However,despiteitsimportance,verylittle
hasbeenstudiedtodetermineindicatorsofperformanceontheexam.Itwasthegoalofthis
projecttolookatundergraduateGPAandMCAT®scores,demographicinformation,medical
schoolexamperformance,andself-reportedsurveydatafromtheUTSouthwesternMedical
SchoolClassof2018todiscovercorrelationsthatmayallowmedicalschoolstobetterpredict
studentperformanceontheexamandguidestudentpreparationfortheexam.Importantly,
thisinformationwouldallowadministratorstoidentifyfuturestudentswhoareatriskoffailing
ordoingpoorlyontheexamandinterveneearlyinordertopreventpoorperformanceonthe
exam.Furthermore,ifstudentsareabletobetterunderstandhowtobestpreparefortheexam
andtobeabletoidentifypersonalredflags,theymaybemorelikelytoindependentlyadjust
studyinghabitsearlyoninordertooptimizetheirperformanceontheexamandultimatelybe
morecompetitiveforresidency.Itisforthesereasonsthatwebelieveitisimportanttoreview
andidentifypredictorsofUSMLEStep1performance.Thisprojecthasthreemaingoals:1)
determiningifdataavailablepriortoadmissioncanpredictStep1performance,2)analyzing
schoolperformanceandsurveydatatosuggeststudyingtips,and3)developingamodeltoflag
studentsatriskofperformingpoorlyontheexamforearlyintervention.
3
BACKGROUND
USMLEStep1isthefirstofthreephysicianlicensureexamstakenbymedicalstudents
andcoversthebasicsciencematerialtaughtduringthepre-clinicalyearsofmedicaleducation.
Itwasdesignedinitiallytoseparatethosewithadequateversusinadequateknowledgeasan
indicatorforpreparednessforclinicaltraining,butithasbecomeatoolusedheavilytostratify
applicants2.Astudent’sscoreonthisexamisconsideredbyresidencyprogramdirectorstobe
thetopfactorinselectingstudentsforinterviews3.OntheUSMLE®Step1exam,itisgenerally
agreedthatascorebelow220willsignificantlydecreaseastudent’schanceofgettingintoatop
residencyprogram,asthisisoftenaminimumscorerequiredtoevenapplytomanyprograms,
eveninlesscompetitivespecialties4.Assuch,itisimportantforbothstudentsandformedical
schoolstodowhattheycantomaximizetheirscorestooptimizetheirchancesofbeing
consideredfortopresidencyprograms.
Startingwiththeadmissionsprocess,medicalschoolsuseMCAT®scoresand
undergraduateGPAsastheirtopfactorsinselectingstudentstointerview,withtheassumption
thathighMCAT®scoresandGPAswillpredictsuccessinmedicalschool,andspecificallysuccess
ontheUSMLE®Step1exam.Anumberofstudiesindicatethatthesepre-admissionsdatamay
bemildpredictorsofStep1scoresbutarewaryoftheweightthatisappliedtothesefactorsin
theadmissionsprocessbecauseofhighvariabilityinpredictivevalue5.Sincethesestudiesall
includeonlymatriculatingmedicalstudents,itispossiblethatthisfactorexhibitsathreshold
4
effect,whereafteracertainpointthepredictivevalueofthesedatapointsbecomesless
significant.Furtherlarge-scalestudiesareneededtobeabletomakeamoreaccurate
assessment.
Thesuccessofapre-clinicalmedicalschoolcurriculumisalsojudgedlargelybythe
performanceofstudentsonStep1,andmedicalschoolprogramsassumethathigh
performanceoninternalexamsindicatethatstudentsarepreparedforStep1.Studentsat
manycampuses,includingUTSouthwestern,areoffereddedicatedstudyingtimeforStep1,
andtherearedozensofresourcesavailabletopreparefortheexam,mostlychosenbyword-of-
mouthrecommendations.Allofthesefactorsplayintoone’sperformanceonStep1,andyet
verylittlehasthusfarbeenstudied.OnlyonestudyatAlbertEinsteinlookedatspecific
resourcesandfoundthatquestionbanksaremorepredictivethanotherresources,andthat
theamountoftimespentstudyinghasnocorrelationwithscores6.
BecauseofthesignificantconsequencesofperformanceonStep1,studiesshowthat
thisexamishighlyrelatedtomedicalstudentburnout.Muchofthisstressarisesfromprograms
teachinginformationthatdoesn’toverlapdirectlywithStep1,with35%ofstudentsreporting
thattheyshifttheirstudyfocusawayfromtheirschools’curricula7.Itwouldthereforebenefit
programstobetterunderstandhowtoguidestudentsinpreparingforthisexamtonotonly
improvescoresonStep1,butalsotoimprovefocusontheschool’suniquecurriculum.Itwould
alsobeusefulforprogramstobeabletoaccuratelyidentifystudentswhoareatriskofpoor
performancesothattheymightbeabletoprovideearlyintervention.Fewstudieshavelooked
5
intodevelopingamathematicalmodel,andevenfewerhavehadsufficientpredictivesuccess,
withonenotableexceptionatMarshallUniversityinDecember20168.Byexpandingonthose
studiesandlookingatawidervarietyofdata,itmaybepossibletoguideotherschoolson
developingindividualizedmodelsthateffectivelyflagat-riskstudents.
DuetotheimportanceoftheUSMLEStep1exam,theprojectaimsinclude:
1. ToidentifyearlypredictorsofperformanceonUSMLEStep1.First,analyzingpre-
matriculationdataofundergraduateGPAandMCAT®scoresanddemographic
factorsmayindicategoodpredictorsofStep1scores.Thisinformationwould
potentiallyallowadmissionsofficestobetterdeterminewhatweightshouldbe
giventothesefactorsindeterminingwhichstudentsareofferedadmission.
2. Toevaluateifsuccessinpre-clinicalcoursesatUTSouthwesterncorrelatetosuccess
onStep1.EvaluatingtherelationofoverallGPAtoexamperformanceand
determiningifspecificcourseorup-trendingordowntrendingscoresthroughout
theyearmayserveaspredictorsforexamscores.
3. Todeterminecorrelationsbetweenspecificresourcesorstudyingstrategiesand
Step1scorestobetterinformstudentsandthemedicalschoolonoptimal
preparationfortheexam.StudentsatUTSouthwesternhavecompletedtwosetsof
surveys–oneduringthededicatedstudyperiodtodetermineintervalscoreson
practicetestsaswellasprogressthroughstudyingmaterials,andthesecondafter
theexamtoretrospectivelydeterminehowmuchtimestudentsspentstudyingand
whichresourcestheyused..Analysisofthesesurveyswillbeperformedtoidentify
6
factorsthatareassociatedwithhigherscoresandmayprovideguidancetofutureat
riskstudents;preventingfailuresorpoorscoresonUSMLEStep1.
4. TodevelopamathematicalmodelthatpredictsperformanceonUSMLEStep1in
orderforadministratorstobeabletoidentifyat-riskstudentsandprovideearly
intervention.
Usingpriorstudiesasaguide,wefirsthypothesizethatpre-admissionsdatawillprovide
onlymildpredictivevalueofStep1scoresandshouldbeusedcautiouslyasadmissionscriteria
andmayexhibitathresholdeffect.Second,performanceinmedicalschoolisdirectlycorrelated
withStep1scores,butdecreasinggradesasstudentsgetclosertotheexamdatewillnot
necessarilypredictperformanceonStep1.Third,studentsusingmoreactiveresourcesand
startstudyingearlywillperformbetterontheexam,butwillseediminishingreturnsaftereach
weekofstudywithscoresmaximizingbyweeks5and6.Finally,thatapredictivemodelcanbe
developedtocategorizestudentsaslow-riskorhigh-risk.
7
METHODS
DataCollectionandSubjectProtection
DatawascollectedfortheUTSouthwesternMedicalSchoolClassof2018(n=238).An
independentdatabrokerwasusedtocompileadatabaseofstudentundergraduateGPA,
MCAT®scores,medicalschooltestscores,ComprehensiveBasicScienceSelf-Assessment
(CBSSA)testscores,anddemographicinformation.Demographicinformationincludedage,sex,
race,yearofundergraduategraduation,priordegrees,concurrentdegrees,leavesofabsence,
andsocioeconomiccategoryonadmission.
Thedatabrokeralsoobtaineddatafrom2surveysthatwerealreadybeingadministered
bytheUTSouthwesternStudentAcademicSupportServicesandtheOfficeofStudentAffairs
viaSurveyMonkey(APPENDIXAandB).Thefirstwasaweeklysurveyduringthededicated6-
weekStep1preparationperiodtodeterminestudent’sscoresontheirpracticetestsand
students’progressintheirstudymaterials.Thesecondsurveywasapost-examsurveywhich
askedstudentswhentheystartedstudying,howmanyweekstheystudied,howmanydaysper
weekandhoursperdaytheystudied,whichresourcestheyused,howmanypracticeteststhey
took,andhowinvolvedtheywerewithextracurricularactivities.Thesesurveyshadstudent
identifiers,whichthedatabrokerwasabletode-identifyusingthesamecodesfromthe
databasedescribedinthepriorparagraphinordertolinkthetwodatabases.Nostudentswere
excludedfromthestudy,butonlystudentswhotookStep1inSummer2016weresurveyed.
Studentswereprovidednoincentivesforsurveycompletion.
8
ThecollectivedatawasassembledbytheresearchteaminanMSExcelspreadsheet
alongwiththestudent’sUSMLEStep1scores.Allstudentswereassignedaparticipantnumber
andtheiridentitieswereprotectedbythe‘databroker’personnelwhomaintainedthe
confidentialrecordoflinksbetweentheuniqueparticipantdataandprojectIDnumber.The
investigatorsandotherlistedteammembersdidnothaveaccesstostudentidentityor
demographicinformation.ThisprojectwasdeemedexemptbytheIRBbecausestudent
identitieswereprotected,anduseofthedatadidnotrequireconsentperFERPA34CFR§
99.319.
Pre-AdmissionsDataAnalysis
ThefirststepofanalysisentailedidentifyingcorrelationsbetweenStep1scoresand
MCAT®,undergraduateGPA,anddemographicdata.Duringthispartoftheanalysis,MSExcel
wasusedtoperformsimplelinearregressiontoidentifyifcorrelationsexisted,withr>0.5
suggestingstrongcorrelation,0.3<r<0.5suggestingmoderatecorrelations,and0.1<r<0.3
suggestingweakcorrelation,perhumanbehaviorstudystandards10.Thiscorrespondedtoour
useofR2>0.25indicatingstrongcorrelationand0.10<R2<0.25indicatingmoderatecorrelation.
P-valuesof<0.05wereconsideredsignificant.
MedicalSchoolGradesandCBSSAScoresDataAnalysis
Asabove,MSExcelwasagainusedtoperformsimplelinearregressiontoidentifyif
correlationsexistedbetweenmedicalschoolgradesorpracticeCBSSAscoresandStep1scores.
Independentgroupst-testswereperformedwithStatPlusinordertobetterunderstand
differencesbetweenthosescoringaboveandbelow220.
9
SurveyDataAnalysis
Fortheweeklyprogresssurveys,simplelinearregressionwasusedinMSExcelto
determineifthereweresignificantcorrelationsbetweenprogressmadeinFirstAid,progress
madeintheQ-bank,andcumulativepercentcorrectintheQ-bankandpracticeexamscoresfor
thatweek.Averagepracticescoresandtheaveragedeltaofpracticescoreswerealsoanalyzed
viasimpleplottingperweekinordertodeterminehowscoreschangedoverthecourseofthe
dedicatedstudyperiod.
Similaranalysiswasperformedonthepost-examsurvey.Toanalyzemedicalschool
gradesandsurveyinformation,linearregressioninMSExcelwasusedtoidentifyifcorrelations
existbetweenStep1scoresandtotalhoursstudied,numberofweeksstudied,hoursof
extracurricularinvolvement,Q-bank%completion,andnumberofpracticeteststaken.
Independentgroupst-testswerealsoperformedtocompareusersofaresourcetonon-users
ofaresourceforeachofthecommonlyusedtest-prepresourcesinordertoidentifyresources
thatresultinhigherStep1scores.
DevelopingaModel
Theprimarygoalofthemodelistopredictifastudentisatriskofscoringunder220on
Step1.Therefore,independentgroupst-testswereperformedinStatPlusinordertoidentify
factorsthatweredifferentbetweenthosewhoscoredbelow220andthosewhoscored220or
higher,withap-valueof<0.05consideredsignificant.Acombinationofthefindingsfromthe
linearregressionmodelsinprioranalysisandtheset-testswereusedtotestthemorehighly
correlatedfactorsviastepwisemultipleregressionanalysisinStatPlusinordertocreatea
predictivemodeltoidentifystudentswhoarelikelytoscorebelowa220.
10
RESULTS
Pre-AdmissionsData
Thedistributionofpre-admissionsscoresforall228studentsintheUT
SouthwesternClassof2018canbereviewedinTable1.EachsectionoftheMCAT®,as
wellasthecombinedbiologicalandphysicalsciencesscoreandthecumulativescore,were
correlatedwithStep1scoresandmedicalschoolgrades(Figures1and2).TheR2values
canbeseeninTable2.Oftheindividualsections,thebiologicalsciencesandphysical
scienceshadR2valuesof0.067and0.095,respectively,indicatingonlyaweakcorrelation
withStep1scores;theverbalreasoningsectionhadanR2valueof0.004,indicatingno
correlation.Thesummationofthebiologicalandphysicalsciencessectionsresultedinan
R2of0.127,indicatingamoderatecorrelationbetweenthesciencesectionsoftheMCAT®
andStep1scores.
Whencorrelatedwithmedicalschoolperformance(cumulativesecondyeargrade
percentage),thecorrelationswithMCAT®scoreswereevenweaker(Figures3and4).
Eventhestrongestpredictor,thesummationofthebiologicalandphysicalsciences
sections,hadanR2valueofonly0.072,indicatingonlyveryweakcorrelationbetween
medicalschoolperformanceandMCAT®scores.
UndergraduateGPAexhibitedsimilartrends,butwereevenlesspredictivethan
MCAT®scores(Figures5and6).AnR2valueof0.104indicatesmoderatecorrelationwith
Step1scores,andanR2of0.058showsonlyminimalcorrelationwithmedicalschool
grades.
11
Ananalysisofdemographicfactorsindicatesthatrace,undergraduatemajor
(sciencevsnon-science),andgenderarenotsignificantlydifferent,asthestep1score
averagesofeachgrouparewithinthemarginoferror(p>0.05foreachrace,maleversus
female,andsciencevsnon-science)(Figure7).Althoughatfirstglanceitseemstherewas
nosignificantdifferenceamongagegroups,afterisolatingonlythoseunderandabove30,
therewasasignificantdifferenceinmeansofalmost8points(237.7vs229.7,p<0.03)
betweenthoseunder30andthose30andolder.Stepscoredifferencesamongthe
socioeconomicgroupswasalsonotsignificant(seeAPPENDIXCforfulldetailsonhow
studentsweresortedintoeachcategory).
MedicalSchoolGradesandCBSSAScores
TheCBSSAexamwasadministeredtoeachstudenttowardstheendofthepre-
clinicalcurriculum,andpriortothededicatedtimeforStep1study.Twohundredelevenof
thestudentsintheclasstooktheexamduringthistimeperiod.Ofthose,theaveragescaled
scoreontheexamwas193(SD±28).TheaveragechangebetweentheCBSSAscaledscore
andtheactualStepScorewas44points(SD±18),witharangefrom-6to+85.Agraphof
Step1scoresversusCBSSAscoresshowsanR2of0.599,indicatingaveryhighcorrelation
(Figure8).Ofallthevariablesstudiedoverthecourseofthisstudy,CBSSAscaledscore
hadthehighestpredictivepoweroffinalStep1scores.
Thesecondmostsignificantpredictivefactorwasmedicalschoolperformance,as
indicatedbythesecond-yearmedicalschoolcumulativegradepercentage(Figure9).217
studentshadcompletedallcoursesandtakentheStep1exam,withanaveragegradeof
84.5%,SD±5.0,rangingfrom69.3%to98.0%.Whensecondyeargradesarebrokendown
byquarter,R2valuesindicatethateachquarterismoreandmorepredictiveofthefinal
12
Step1scores,withtheexceptionofthefinalquarter,whichisfarlesspredictivethanthe
otherthree,butstillshowingastrongcorrelationwithStep1scores(Table3).
Therewasverylittlechangeinscoresbetweeneachquartergradingterm,bothfor
individualsandonaverage(Table4).However,whenplottingjusttheindividualchange
betweenthethirdandfourthquartergradescomparedtoStep1scores,individualswhose
scoresdecreasethemostbetweenthethirdandfourthquartersendupwithhigherStep1
scores(Figure10).AnR2of0.137onthisplotindicatesthatthereisamoderatenegative
correlationbetweenthosewhosegradesdropbetweenQ3andQ4andtheirStep1scores.
Thissamerelationshipisnotseeninthechangebetweenthesecondandthirdquarters.
SurveyData
SASSWeeklySurvey
WeeklysurveysweredistributedduringthededicatedStep1preparationperiod
andadministeredbytheStudentAcademicSupportServices.Thesesurveyshad198
uniquerespondersovera6weekperiod,with174respondersinweek1,taperingto43
respondersinweek7.Becauseidentifierswereused,wehadtheabilitytotrackthesame
individualstudentresponsesoverthe7-weekperiodthatthesurveyswereadministered.
Themostpredictivedatapointonthesesurveyswasthecumulative%correctin
theUWorld™and/orUSMLERx™questionbank,thetwoquestionbankresourcesusedby
everystudentsurveyed.Aplotofthe%correctversusthepracticeexamscoreforthat
weekshowsanR2of0.507,indicatingastrongcorrelation(Figure11).Lesspredictiveand
moderatepredictorsweretheprogressmadethroughtheQ-bank(s)andFirstAid™,
respectively,withR2valuesof0.241and0.200(Figures12and13).Thepercentages
13
includemultiplepassesbutdon’tdistinguishbetweendifferentresources,sosomeonewho
washalfwaythroughasecondpassoftheU-World™Q-bankwouldhave150%,aswould
someonewhocompleted100%oftheU-World™Q-bankand150%ofUSMLERx.
EachconsecutiveweekofstudyproducedsmallergainsinpointsontheUSMLE1.
Figure14showsthatingeneral,scorescontinuetoincreasethroughoutthefirst6weeks
ofstudy,butlessandlesseachweek,withasignificantdropbyWeek7.Itisimportantto
notethatstudentswereonlygiven6weeksofdedicatedpreparationtime,sothosewho
spentlongerthan6weekshadtorequestextratime.Forthefirst6weeks,aquadraticline
ofbestfitwasdeterminedtobe:
StepScore=-0.4365x(#weeks)2+8.5953(#weeks)+201.16
withanR2of0.972,indicatingaverystrongfittothedata.Thisequation’svertexis243.56
at9.87weeks,indicatingthatbetweenweeks6and10,astudentmaypotentiallygain6
points,orlessthan2pointsperweekofstudy.
Figure15includesonlydatapointsforstudentswithtestscoresintwoconsecutive
weeks,andshowsthatwhilestudentscouldgainanaverageof13.9pointsinthefirstweek
(SD15.1),thegainsdropprogressivelyweek-to-week,withanaverageincreaseofjust4.1
pointsbetweenweeks5and6(SD12.0),andmanystudentsnotincreasingoreven
droppingscoresatthatpoint.Thedifferencebetweenweeks6and7wasnotplotted
becausetherewereonly5studentswhosubmittedbothweek6andweek7practiceexam
scores.Alinearbest-fitlineonthisgraphindicatesthatafterweek6,onaveragenofurther
scoreincreasesshouldbeexpected,withanR2valueof0.938indicatingthestrong
predictivestrengthofthismodel.
14
Post-ExamSurvey
Thepost-examsurveyadministeredwithinafewweeksoftheStepexamhad140
responsesoutof228totalstudentsintheclass.Thedistributionofscoresofthosewho
completedthesurveycomparedtotheentireclassisshowninTable5.Thosewho
completedthesurveyscored,onaverage,5pointshigherthantheclassasawhole,witha
lowerstandarddeviationamongsurveyrespondentsindicatingthatasignificant
proportionofthosewhodidn’tcompletethesurveyperformedworseontheexam.
Acomparisonofusersversusnon-usersofeachresourceindicatesthatthosewho
useKaplan™,NBME®practiceexams,UWorld™SelfAssessments,BRS™Physiology,and
USMLERx™scoredhigher(Figure16).ThosewhousedDoctorsinTraining™andSketchy
Pharm™scoredlower.Inthisstudy,onlySketchyPharm™hadp<0.05,sothedifferences
seeninthegrapharenotstatisticallysignificant.
ThenumberofquestionsansweredineithertheUWorld™,USMLERx™,orKaplan™
questionbankhadamildtomoderatecorrelationwithanR2of0.094.However,
consideringthatUWorld™is~2350questions,groupingthedataintothoseanswering
fewerthan50%oftheQ-bankquestions,thoseanswering50-150%,andthoseanswering
greaterthan150%,thesignificanceofdifferencebecomesmoreapparent,especiallyatthe
higherend(Figure17).Studentswhoansweredfewerthan1700questionsscoredan
averageof230±10.6(95%CI),thosewhoanswered1701-4000questionsscoredan
averageof241±2.4(95%CI),andthosewhoansweredgreaterthan4000questions
scoredanaverageof254±5.3(95%CI).
StudentswhostartedstudyingforStep1earlierperformedbetter,althoughthedata
doesn’tsuggestaclearidealstarttime,sincethereisnosignificantdifferencebetweenany
15
oftheadjacentcategoriesbeginning1year,6months,3months,or6weekspriortothe
exam(Figure18).Thosewhostartedtostudy1yearpriortotheexamscoredonaverage
254±8.5(95%CI),thosewhostarted6monthspriorscored252±6.9(95%CI),those
whostarted3monthspriorscored242±3.2(95%CI),andthosewhowaiteduntilthe
dedicated6-weekpreparationtimescored237±3.6(95%CI).
Whenaskedhowmanyhoursstudentsstudiedwithinthe6-weekpreparationtime,
therewerenosignificantdifferencesfoundintermsofnumberofweeksspentstudyingor
totalhoursstudyingoverthecourseofthe6weekperiod(Figures19and20).Statistical
erroraccountedfordifferencesineverycategory.
Somewhatsurprisingly,extracurricularinvolvementappearstohavenoimpacton
Step1scores(Figure21).Infact,thosewhostartedtostudy1yearand6monthspriorto
theexamhadhigherlevelsofextracurricularinvolvementat7.7hoursand8.8hours,
respectively,comparedtothosewhostartedstudyinglaterat6.0and6.3hours.Therewas
alsonosignificantdifferenceintheextracurricularinvolvementandtotalhoursstudied
duringthededicatedstudyperiod.
Studentswereaskedtonamethelastpracticeexamtheytookandthescorethey
receivedpriortotakingtheStep1exam.Figure22showsthatNBME®18isanaccurate
andprecisepredictorofone’sfinalscoreonStep1,withanaveragedifferenceof0.06
points±2.0.UWorld™SelfAssessment2(+2points±5.0,95%CI)andNBME®15(-2.5
points,±7.2)areaccuratebutlessprecise.NBME®13(+5.5points,±4.8,95%CI)and
NBME®17(-4.5points,±3.4,95%CI)arelessaccuratepredictors,andalsonotasprecise.
NBME®14(+1point,±13,95%CI)isimprecise.NBME®16(+3points,±2.9,95%CI)isless
preciseandaccuratethanNBME®18,butisotherwiseagoodpredictor.
16
DevelopingaModeltoPredictStep1Performance
Todevelopamodeltoidentifythoseatriskofscoringbelow220onStep1,t-tests
wereperformedforanumberofcategoriesthatsuggestedcorrelationinthepre-
admissionsdatathesurveys,andthemedicalschoolgradesandCBSSAscoresforthose
studentsscoringbelow220comparedtothosescoring220orhigher.Table6showsthe
averages,standarddeviation,andp-valuesobtainedfromthet-testsforeachtested
variable.
Thet-testsindicatedthat6categorieshadsignificantdifferencesbetweenthe
groups:undergraduateGPA,MCAT®®®score,#ofquestionsansweredintheQ-Bank,total
hoursstudying,CBSSAscore,andcumulativeMS2grades.
Basedontheaveragesandstandarddeviationsshown,aroundedthresholdroughly
1-2standarddeviationsbelowtheaverageforthe<220groupwaschosenandapositive
predictivevalueforselectingastudentwhowouldscore<220foreachofthetestswas
calculated(Table7).Mostnotably,astudentwithaCBSSAscoreoflessthan160hasa64%
chanceofscoringbelow220,andastudentinthebottom15%oftheclasshasa60%
chanceofscoringbelow220.ThenextmostpredictivefactorsweretheundergraduateGPA
andMCAT®scores,withanundergraduateGPAlessthan3.5predictinga45%chanceof
scoringbelow220,andanMCAT®lessthan30witha48%chance.Theleastpredictive
wereansweringfewerthan2000Q-bankquestionsorspendinglessthan250totalhours
studying,withapositivepredictivevalueof27%and30%,respectively.
Usingthesefindings,aseriesofmultiplelinearregressionsweretested,usinga
mixtureofcontinuousvariablesandcategoricalvariablesasseenonTable7.Sincethe
purposeofthemodelwastoflagstudentsatriskofperformingpoorlyearly,thenumberof
17
Q-bankquestionsansweredandtotalhoursstudyingwereleftoutofthemodel,asthose
arefactorsthatwouldn’tbeknownuntiltheendofthestudyperiod.Sincetherewasa
significantdifferenceinMCAT®scoresandundergraduateGPAsusingathreshold,those
factorswereleftasconstantseitheraddedorleftoutofthemodeltoincreasethe
predictivevalue.Finally,theinitialCBSSAandcumulativesecondyeargradepercentage
hadsuchstrongcorrelationsthatthosewereincludedwithlinearcoefficients.Theresultof
themultiplelinearregressionwasthefollowingmodel:
USMLE1Score=100.84962+4.85204(ifUndergraduateGPA<3.5)+3.66261(ifMCAT®<31)+0.31215*[CBSSAScore]+0.8223*[CumulativeMS2Grade%]
TheregressionstatisticsareshowninTable8.UndergraduateGPAandMCAT®
scoresdidnothavesignificantp-valueswhenrunthroughthemodel,buttheyimproved
theoverallpredictivevalueofthemodel,sotheywerehelpfultoinclude.AnR2valueof
0.64indicatesthatthismodelaccountsfor64%ofthevariabilityinthepopulation,anda
standarderrorof8.87indicatesthatthismodelwillpredict67%ofscoreswithin8.87
points,and95%ofscoreswithin17.74points.
Whentestingthismodeltoidentifythoseatriskofscoringbelow220,thestandard
errorwasusedastheupperendofthemodel.Thatis,anystudentpredictedbythemodel
toscore228.87orlowerwasconsideredapositivetest.Whenappliedtothegroupasa
whole(186studentshadalldatapointsnecessarytorunthismodel),thisresultedina
sensitivityof81%(25/31),specificityof86%(134/155),positivepredictivevalueof54%
(25/46),andnegativepredictivevalueof96%(134/140).Thatis,eventhoughthemodel
onlypredicts67%ofscoreswithinthestandarderror,usingthistoflagstudentscould
18
catch81%ofthosewhoultimatelyscoredbelow220.Ifastudentwasfoundtoscore
higherthana228.87bythemodel,thenthereisa96%chancethatthestudentwillnot
scorebelow220.Ofnote,38%ofthefalsepositives(8/21)scored225orbelow.
19
DISCUSSION
Pre-AdmissionsData
AlthoughsignificantweightistraditionallyplacedonMCAT®andundergraduate
GPAs,clearlyonlyamild-to-moderatecorrelationexistsbetweenStep1scoresandMCAT®
scores,onlyweakcorrelationsbetweenStepscoresandundergraduateGPAsandeven
weakercorrelationsbetweenMCAT®orGPAstomedicalschoolperformance.Theverbal
sectionoftheMCAT®isparticularlypooratpredictinganylevelofperformanceinmedical
school.However,thesetwofactorsarestillthebestmeasuresavailabletoeasilysort
throughapplicants.Infact,afterputtingtogetheramodeltopredictperformance,itisclear
thatthosewithMCAT®scores30orbelow,orGPAslowerthan3.5areatasignificantrisk
ofperformingpoorlyonStep1.ThisdatamaythereforebeskewedbythefactthatUT
SouthwesternstudentshavegenerallyveryhighMCAT®scoresandundergraduateGPAs.
Thisindicatesathresholdeffect–ifastudentscoresabovea30ontheMCAT®andhas
greaterthana3.5GPA,perhapsnoadditionalconsiderationshouldbemadewithregards
tothosescores.However,thisdatamaysuggestthatadmissionspersonnelbewaryof
studentswhodonotmeetthosethresholds.
Race,undergraduatemajor,socioeconomicclass,age,andgenderhavenosignificant
effectonStep1scores,withanydifferencesofaverageswellwithinthemarginoferror.
Studentscertainlyreportbenefitingfromadiverseclassintheannualgraduation
questionnaire,andampleresearchsuggeststhemanybenefitsofdiverseexposuresand
experiences11.Admissionscommitteesmayconsiderusingthesefactorstohelpcreatea
20
diverseclasstoimprovethelearningatmosphereasawhole,butdemographicfactorshave
nobearingonhowastudentislikelytoperformonStep1.
MedicalSchoolGradesandCBSSAScores
ItisclearthatmedicalschoolgradesandCBSSAscoresarethestrongestpredictors
ofStep1performance,bothshowingveryhighcorrelations.Althoughmanystudentshave
atendencytoshifttheirfocusawayfrommedicalschoolexamsastheybegintopreparefor
Step1,itshouldbemadecleartostudentsthat,atleastatUTSouthwestern,doingwellin
courseswilltranslatedirectlytoperformingwellonStep1.
CBSSAscoresarethesinglehighestpredictorofStep1scores.Thisiseasyto
understand,asthisexamisthesameformatandstyleoftheactualexam,andisafair
assessmentofthebaselineknowledgeofamedicalstudentbeforegoingintoadedicated
preparationperiod.Becausestudentsareallgivenessentiallythesameamountoftime,it
makessensethatthosewhostartaheadwilltendtostayahead.Thatsaid,thereis
significantvariationinhowmuchstudentscanlearnoverthat4-6weekperiod,withsome
studentsincreasingtheirCBSSAscoresbyasmuchas85points.Thosescoringbelow160
(scaledStepscore)haveasignificantriskofdoingpoorlyontheexamandmaywarrant
moreinterventionearlyon.
Itwassurprisingtoseethatstudents’gradesdidnotvarysignificantlybetweenQ2
(Quarter2)andQ3,andevenstudentswhosegradesdecreased,presumablybecausethey
startedtoshiftfocustowardsstep,didnotperformbetteronStep1.Thatwasnottrue
betweenQ3andQ4.OutofallofthesecondyeargradesQ4gradeswerebyfartheleast
predictiveofStepscores,andstudentswhodroppedbetweenQ3andQ4tendedtoscore
higheronStep1.Thismaybeexplainedbyselectionbiasandconfoundingfactors.Students
21
whocouldaffordto“slackoff”anddroptheirQ4gradesweremostlystudentswhowere
notalreadystruggling.Furthermore,bytheendofQ3,UTSouthwestern’scurriculumhad
coveredabulkofthecoreStep1material,withonlydermatology,psychiatry,and
integratedsystemslefttocoverinQ4.Becausethosetopicsareonlyasmallpercentageof
whatiscoveredonStep1,itispossiblethatstudentsbenefitedbystudyingmorehigh-yield
Step1topicsduringthatquarter,whichcouldresultinhigherscores.Thesearethesame
studentswhowouldbeexpectedtohavestartedstudyingbetween3monthsand6months
priortotheexam,whichwasalsocorrelatedwithhigherStep1scores.
Overall,theconclusionremainsthatdoingwellonmedicalschoolexamswill
translatetodoingwellonStep1,somethingadministratorsandeducatorswillappreciate.
However,itispossiblethatasonegetsclosertotheexam,theremaybesomebenefitto
shiftingfocusawayfromschoolandtowardstheexam,particularlyiftheschoolcurriculum
leaveslower-yieldtopicsfortheendoftheyear.
SurveyData
Althoughthesurveyparticipantsscoredhigherthantheclassasawhole,the
surveysprovideusefulinformationtoguidepreparationstrategies.
Inthecomparisonofusersofresourcesversusnon-users,noneofthedatapoints
werestatisticallysignificantexceptthatusersofSketchyPharm®performedsignificantly
worsethannon-users.However,thisislikelyaresultofthepowerofthisportionofthe
study.Ingeneral,itcanstillbeconcludedthatusersofquestion-basedresources–Kaplan™
QBank,NBME®assessments,UWorld™SelfAssessments,andUSMLERx™–performed
betteronexamsthannon-users.Thismayindicatethatthebestpreparationstrategiesare
toanswerasmanyquestionsaspossible.Thisisfurthersupportedbythefactthatthose
22
whoansweredmorethan4000Q-Bankquestionsoutperformedthosewhoanswered
fewer.UsersofthetwovideoresourcesDoctorsinTraining™andSketchyPharm™were
theworstperformers,whichmaybeduetothefactthatvideoresourcestakealongtimeto
gothroughandimpedetheabilitytodomorepracticequestions.
Lookingattheweeklysurveys,thefactorsmostcorrelatedwithhighpracticeexam
scoresarethecumulative%correctintheQ-bankandprogressthroughtheQ-bank,with
progressthroughFirstAid™alsocorrelatedwithhigherscoresbutlessso.Thissupports
theassertionthatquestionbankresourcesarethehighest-yieldresourcesinpreparation
forthisexam.
Thisdataalsoprovidesveryusefulinformationinguidingstudentsonwhentostart
preparingandforhowlong.StudentsatUTSouthwesternwereallgiven6weeksofffor
dedicatedstudy,andmoststudentsdidn’tstartintensestudyinguntilthistime.However,it
isclearthatthosewhostartedstudying1yearor6monthspriortotheexamperformed
betterthanthosewhostartedstudyingeither3monthsprioror6weeksprior.Thatsaid,
oncethededicatedstudyperiodstarted,theresultswerelessclear,astherewasno
significantdifferenceinscoresamongdifferentweeksstudyingortotalhoursstudying
duringthededicatedpreparationperiod.Theaveragescoresforthosewhostudiedfewer
than250totalhoursandthosewhostudiedover600hourswerewithinthemarginof
error,alongwitheachintervalinbetween.Moreimportant,itseems,wasthelevelatwhich
studentscameintothepreparationperiod.Thatsaid,therewasasignificantdifferencein
hoursstudiedbetweenthosewhoscoredbelow220andthosewhoscored220orhigher,
soitwouldnotbeunreasonabletosuggestthatstudentsshouldstillaimtostudy300
hours,whichis1standarddeviationbelowtheaverageforthosewhoscored220orhigher.
23
Thiscorrelatestoroughly50-60hoursperweekfor5-6weeks.Ofnote,thiswas
retrospectivedataandmaynotaccuratelyrepresenttheexacttimestudying,asitmay
includetimespentonbreaks,stretching,eating,etc.
Thisnumberisfurthersupportedbytheweeklysurveys.Itisclearthatstudents
havediminishingreturnsaftereachweekofstudy.Fittingaquadraticfunctiontothefirst
sixweeks(week7wasanoutlierbecausethesewerestudentswhorequestedextratime
andwerethelowestperformers)suggeststhatafterWeek6,studentsshouldexpect,on
average,nomorethan1-2pointsperweek,andeveniftheystudy10weeks,theyare
unlikelytogetmorethan6points.Bylookingatjustthedelta,thelineofbestfitiseven
moreconservative,statingthatstudentsareunlikelytomakeanyincreaseinscoresafter
week6.Ofcourse,thereareexceptions,butformoststudents,itappearsthatscoresmay
starttoeitherstabilizeorevendecreaseafterweek6.Giventhisdata,itappearsthat5-6
weeksisoptimalfordedicatedpreparationtimeamongstudentsatUTSouthwestern.
DevelopingaModeltoPredictStep1Scores
Therewereanumberoffactorsthatdidnotshowstrongcorrelationswhenlooking
attheentiregroupofstudents,butwhenonlycomparingthosewhoscored<220tothose
whoscored220orhigher,afewmoresignificantrelationshipsemerge.Undergraduate
GPAsandMCAT®scores,asexpected,weresignificantlydifferentbetweenthetwogroups.
Alsosignificantwerethenumberofquestionsansweredandthetotalhoursstudying,both
withpvalues<0.02.Thissuggeststhatalthoughonthewholethecorrelationmaybeweak,
thosewhoperformbettermeetatleastacertainthresholdofstudying.Theothernotable
significantfactorsaretheCBSSAscoresandcumulativeMS2grades,alsoasexpectedbased
onthepriordata.
24
Themodeldevelopedcanserveasaveryusefulresourceforeducatorstoflag
studentsatriskofpoorperformanceonStep1,definedasscoringbelowa220.Itusesa
thresholdforundergraduateGPAandMCAT®scores,withpointsdeductedifastudenthas
lowerthana3.5orscores30orbelowontheMCAT®,anditappliesacoefficienttothe
students’CBSSAscoreandmedicalschoolgrades.TheundergraduateGPAandMCAT®
don’tindividuallyhavelowp-valuesinthemodelandcontributeminimallycomparedto
theCBSSAandmedicalschoolgrades,buttheydostillimprovethepredictivevalueofthe
model.
Themodelisabletopredict67%ofscoreswithin8.87points,and95%ofscores
within17.39points.Moreimportantly,though,itcanserveasagoodwaytoflagstudents
atriskofscoringpoorlyontheexam.Ifweusethe67%confidenceintervalandmarkthe
upperlimitofthemodelat228.87,itwillcatch81%ofthosewhowouldscorebelow220
with86%specificity.Italsohasanegativepredictivevalueof96%,soitcanalsoalleviate
concernsaboutamajorityoftheremainderofthestudentbody.Ofnote,althoughthe
modelcanserveasausefulresourceforeducatorsandadministratorstoflagat-risk
students,individualstudentsshoulduseitwithcaution.Someofthosewhoarepredicted
toscorelowmayverywelldoverywellontheexam,justasthosepredictedtoperform
wellmaydopoorlyontheexam.Themodelisausefulguide,butitdoesnotpredictdestiny.
25
CONCLUSION
Thisprojectdemonstratesthatobjectivedataanalysiscanbeusedtoexplain
studyingstrategiesandtoguidestudentstowardsoptimalpreparationforStep1.Thesame
datacanbeusedbyeducatorstoflagstudentsatriskofpoorperformancetointervene
early.
Thefirstaimoftheprojectwastoanalyzepre-matriculationdata.AtUT
Southwestern,itwasshownthatmostpre-matriculationdataserveasonlyweak
predictorsofmedicalschoolorUSMLE®examperformance,andthatdemographic
informationhasnoinfluenceonperformanceinmedicalschool.However,thereisacertain
thresholdthatindicatesbasiccompetence,underwhichstudentswillbeatriskofpoor
medicalschoolperformance.Thissuggeststhatmedicalschooladmissionsofficersshould
useGPAandMCAT®scorestomakesurestudentsreachacertainthreshold,butbe
cautiousaboutplacingtoomuchweightonthesecharacteristicsbeyondthatpointandlook
insteadtootheraspectsoftheapplications.Futurestudiesinthisareacouldfurther
explorethepredictivepowerofMCAT®andGPAsforamorepowerfulstudy,andalso
exploreotheraspectsthatmaybeusefulinsortingapplications.
Thesecondaimoftheprojectwastodetermineifperformanceinmedicalschool
correlatedtoperformanceontheStep1exam.Itisabundantlyclearthatmedicalschool
gradesaresomeofthestrongestpredictorsofStep1performance,sostudentswho
performwellinmedicalschoolshouldbeexpectedtoperformwellonboardexams.
However,itisimportanttonotethesmallcaveatshowninstudentswhoshiftedtheirfocus
inthelastfewmonthsoftheschoolyearduringlesshigh-yieldtopics–thosewhosescores
26
wentdowninthelastquarterperformedbetteronStep1.Althoughtheimplicationsofthis
dataislimitedatthispoint,asthisclasswasthefirsttoswitchtopass/failatUT
Southwesternandthecurriculumhasbeensincereformedforfutureclasses,this
highlightstheimportanceofensuringthatthemedicalschoolcurriculumremainshighly
relevantthroughoutthepreclinicalyears,andperhapspointstowardsaneedformore
Step-focusedcourseworktowardstheendofthepre-clinicalyears.
Thethirdgoalofthisprojectwastoidentifyifthesurveysmightprovideuseful
studyingtips.Thedataindicatesthatthebestwaytopreparefortheseboardexamsistogo
throughasmanypracticequestionsaspossible.Thedataalsoindicatesthatstudying
earlierwillleadtohigherscores,butthatoncethededicatedpreparationperiodhasbegun,
5-6weeksseemsoptimal.Twodistinctmodelsbothshowthatscoreswillincreaseonly
minimally,ifatall,afterthe5-6weekmark,sostudyingfurtherthanthatamountwouldbe
apooruseoftime.Furtherstudiesshouldlookintonewresources,aswellasaimformore
powerforeachresourceinordertobetterdeterminewhatthebestresourcestrulyare.
Finally,thisprojectaimedtodevelopamodeltocatchthoseatriskofscoringbelow
220onStep1.UtilizingathresholdofundergraduateGPAandMCAT®scores,anda
coefficientmultiplierformedicalschoolgradesandinitialCBSSAscores,amodelwas
developedwithsignificantpredictivepotential.Thisprojectshowsthatamathematical
screeningtoolmightbeabletoresultintangiblesuccessbyvirtueofpreventingfailurefor
asignificantnumberofatriskstudents.Nextstepsincludevalidatingthismodelacross
differentclassesatUTSouthwesternandacrossmultiplecampusesbeforeitcanbe
confidentlyusedtostratifystudents,butitatleastprovidesguidanceonwhatmedical
schooleducatorsandadministratorsshouldlookforastheypreparestudentsforStep1
27
exams.Futurestudiesshouldalsodetermineifearlyinterventionwillimprovetheoutcome
forat-riskstudents,becauseitispossiblethatthesestudentsmaynotperformsignificantly
higherevenwithappropriateinterventions.
28
LISTOFTABLES
Table1.DistributionofPre-AdmissionsScoresforUTSouthwesternClassof2018.
Variable Average StandardDeviation
MCAT®Score 33.79 2.94
Undergraduate
GPA
3.83 0.18
Table2.R2valuesforMCAT®sectionscomparedtoStep1scoresandMedicalSchoolGrades
MCAT®Section R2forStep1Scores R2forMedicalSchoolGrades
BiologicalScience(BS) 0.06717 0.06577
PhysicalScience(PS) 0.09543 0.02945
VerbalReasoning(VR) 0.00041 0.00081
BS+PS 0.12715 0.07188
TotalScore 0.10168 0.05981
Table3.CorrelationstatisticsforsecondyearquartertermgradescomparedtoStep1scores.
GradingTerm R2forStep1Scores
Quarter1 0.4425
Quarter2 0.45742
Quarter3 0.4886
Quarter4 0.29051
Cumulative 0.52221
29
Table4.DistributionandcorrelationstatisticsforgradechangebetweensecondyearquartertermgradescomparedtoStep1scores.Terms Average
Grade
Change(%)
Standard
Deviation
R2forStep1Scores
Q1-Q2 -2.36 3.32 0.13717
Q2-Q3 1.48 3.26 0.0088
Q3-Q4 2.81 3.52 0.0184
Table5.DistributionofscoresofsurveypopulationversusactualpopulationofUTSouthwesternClassof2018 Survey Actual
NumberofStudents 140 228
Average 242 237
StandardDeviation 14 17.5
Min 202 169
Max 272 272
30
Table6.Distributionandt-testsstatisticscomparingstudentswhoscored<220versusthosewhoscored220orhigheronStep1 <220 220+
Category AVG SD AVG SD p-value
UgrdGPA 3.71 0.29 3.83 0.184 0.01741MCAT® 31.77 2.94 34.15 2.55 0.00003#Questions 2173.33 568.48 2527.49 686.76 0.01838#WeeksStudying 5.73 0.729 5.73 0.928 0.4937#DaysperWeek 6.47 0.48 6.54 0.6 0.28827TotalHoursStudying 350 109 420 120 0.01804PassesThroughFirst
Aid
2.09 0.99 2.46 1.02 0.097
NumberofPractice
ExamsTaken
4.2 1.47 4.75 1.47 0.09245
CBSSAScore 159 14 199 25 0Changebetween
NBMEandStep
scores
46.4 15.8 43.2 18.5 0.15894
CumulativeMS2
Grades%
78.98 3.84 85.51 4.52 1.42E-11
31
Table7.Positivepredictivevalueforscoring<220onStep1forvariousriskfactors
RiskFactorsofScoring<220
Postitive
Predictive
Value
UndergraduateGPA<3.5 45%
MCAT®<31 48%
<2000Q-bankquestionsanswered 27%
<250totalhoursstudied 30%
InitialCBSSAscaledscore<160 64%
CumulativeGrades<80%(inbottom15%of
class) 60%
32
Table8.RegressionstatisticsformultiplelinearregressionmodelforpredictingUSMLEStep1scoresRegressionStatistics
R 0.80236
R-square 0.64379
StandardError 8.86639
N 142
USMLE®Step1Score=100.84962+4.85204(ifUndergraduateGPA<3.5)
+3.66261(ifMCAT®<31)+0.31215*[CBSSAScore]+0.8223*
[CumulativeMS2Grade%]
Coefficient StandardError p-level H0(5%)
Intercept 100.84962 14.81136 2.83E-
10
rejected
UndergraduateGPA 4.85204 4.70029 0.30376 accepted
MCAT®Score 3.66261 2.98453 0.22185 accepted
CBSSAScaledStep1Score 0.31215 0.04042 2.15E-
12
rejected
CumulativeMS2Grade% 0.8223 0.22227 0.00031 rejected
33
LISTOFFIGURESNote:Errorbarsonallgraphsindicatea95%confidenceinterval.
Figure1.CorrelatingUSMLEStep1scoreswithMCAT®Scores.R2indicatesmoderatecorrelation.
Figure2.CorrelatingUSMLEStep1scoreswithsumofBiologicalSciencesandPhysicalSciencesMCAT®Scores.R2indicatesmoderatecorrelation.
R²=0.10168
160
180
200
220
240
260
280
25 27 29 31 33 35 37 39 41 43
Step1Score
MCATScore
Step1ScorevsMCAT
R²=0.12715
160
180
200
220
240
260
280
15 17 19 21 23 25 27 29 31
Step1Score
MCATBS+PSScore
Step1ScorevsMCATBS+PS
34
Figure3.CorrelatingsecondyearmedicalschoolgradeswithMCAT®Scores.R2indicatesweakcorrelation.
Figure4.CorrelatingsecondyearmedicalschoolgradeswiththesumofBiologicalSciencesandPhysicalSciencesMCAT®Score.R2indicatesweakcorrelation.
R²=0.05981
6065707580859095100
25 27 29 31 33 35 37 39 41 43
MS2Cum
ulativeAverage(%
)
MCATScore
MS2GradesvsMCAT
R²=0.07188
6065707580859095100
15 17 19 21 23 25 27 29 31
CumulativeGrade%
MCATBS+PSScore
MS2GradesvsMCATBS+PS
35
Figure5.CorrelatingStep1scoreswithundergraduateGPA.R2indicatesmoderatecorrelation.
Figure6.CorrelatingsecondyearmedicalschoolgradeswithundergraduateGPA.R2indicatesweakcorrelation.
R²=0.10355
160
180
200
220
240
260
280
2.60 2.80 3.00 3.20 3.40 3.60 3.80 4.00
Step1Score
UndergraduateGPA
Step1ScorevsUndergraduateGPA
R²=0.05751
6065707580859095100
2.60 2.80 3.00 3.20 3.40 3.60 3.80 4.00
MS2Cum
ulativeAverage(%
)
UndergraduateGPA
MedicalSchoolGradesvsUndergraduateGPA
36
Figure7.AverageStep1scoressplitbydemographicfactors.(A)showsaveragesamongdifferentraces,(B)showsscienceversusnon-sciencemajors,(C)showsaveragesamongdifferentagegroups,(D)showsaveragessplitbygender,and(E)showssplitbasedonsocioeconomiccategories,withAbeinglowsocioeconomicstatusandDindicatingaffluentfamilybackground.
200
220
240
260
White Asian Hispanic BlackAverageStepScore
Race
RaceandStepScores
225230235240245
Science NotScience
AverageStep1Score
UndergraduateMajor
UndergraduateMajorvsStepScore
210220230240250
<26 26-29 30+
StepScore
Age
EffectsofAgeonStepScore
225230235240245
M F
AverageStepScore
Gender
EffectsofGenderonStepScores
180200220240260
A B C D
AverageStep1Score
SocioeconomicClass
EffectsofSocioeconomicClassonAverageStepScore
A B
C D
E
37
Figure8.CorrelatingStep1scorewithinitialCBSSAscaledscore.R2indicatesverystrongcorrelation.
Figure9.CorrelatingStep1scorewithsecondyearmedicalschoolcumulativegradeaverage.R2indicatesverystrongcorrelation.
R²=0.59936
160180200220240260280300
120 140 160 180 200 220 240 260 280
Step1Score
InitialCBSSAScore(PredictedStepScore)
Step1ScorevsCBSSAScore
R²=0.52221
160180200220240260280
65 70 75 80 85 90 95 100
Step1Score
MS2CumulativeGradeAverage(%)
Step1ScorevsMS2CumulativeGradeAverage
38
Figure10.Correlationbetweenthechangeingradesbetweenthirdquarterandfourthquartergradingtermsandastudent’sStep1score.ThissuggeststhatstudentswhosegradesdropbetweenQ3andQ4performbetterontheexam,withR2indicatingmoderatecorrelation.
Figure11.Correlatingsurveydataofcumulative%correctintheUWorld,Kaplan,orUSMLERxQ-bankswithpracticeexamscoresinthesameweek.R2indicatesstrongcorrelation.
R²=0.13717
160
180
200
220
240
260
280
-10 -5 0 5 10 15
Step1Score
DeltaQ3toQ4(%points)
Step1ScorevsChangeinQ3toQ4Grades
R²=0.50715
140160180200220240260280
30 40 50 60 70 80 90 100
PracticeExamScore
Cumulative%CorrectinQ-Bank
PracticeExamScoresvsCumulative%CorrectinQ-Bank
39
Figure12.CorrelatingsurveydataofprogressthroughUWorld,Kaplan,and/orUSMLERxQ-bankwithpracticeexamscoresinthesameweek.150%canindicateeitherfinishingtheQ-bankonceandrepeating50%,orfinishing1fullQ-bankand50%ofanother.R2indicatesmoderate/strongcorrelation.
Figure13.CorrelatingsurveydataofprogressthroughFirstAidwithpracticeexamscoresinthesameweek.150%canindicateeitherfinishingtheQ-bankonceandrepeating50%,orfinishing1fullQ-bankand50%ofanother.R2indicatesmoderatecorrelation.
R²=0.24149
130150170190210230250270290
0 50 100 150 200 250
PracticeExamScore
ProgressThroughQ-Bank(%)
PracticeExamScoresvsProgressThroughQ-bank
R²=0.19987
130150170190210230250270290
0 50 100 150 200 250 300 350 400 450 500
PracticeExamScore
ProgressThroughFirstAid(%)
PracticeExamScoresvsProgressThroughFirstAid
40
Figure14.Surveydatashowingaveragepracticeexamscoresforeachweekofthesurvey.Aquadraticbest-fitfunctionforjustweeks1through6issuperimposed,withtheR2indicatingveryhighcorrelationbetweenthemodelequationandthedata.Week7wasanoutlierbecausethereweresofewstudents,allofwhichspecificallyrequestedextratimetostudy.
y=-0.4365x2+8.5953x+201.16R²=0.97238
180
190
200
210
220
230
240
250
260
1 2 3 4 5 6 7
PracticeExam
Scores
WeeksofStudy
AveragePracticeExamScoresover7weeksofStudy
41
Figure15.Surveydatashowingthechangeinscoresfromoneweektothenext.Onlyconsecutivedatapointswereincluded.Alinearbest-fitfunctionforweeks1through6issuperimposed,withtheR2indicatingveryhighcorrelationbetweenthemodelequationandthedata.Week7wasanoutlierbecausetherewereonly5studentswithconsecutivedatapoints,allofwhichspecificallyrequestedextratimetostudy.
13.88
11.79
8.36
3.86 4.08
y=-2.7525x+16.652R²=0.93793
0
5
10
15
20
25
Weeks1-2 Weeks2-3 Weeks3-4 Weeks4-5 Weeks5-6
ChangeinPracticeExam
Scores
WeeksofStudy
AverageΔinPracticeScoresBetweenWeeks
42
Figure16.Averagescoresofusersversusnon-usersofeachresource.OnlySketchyPharmhadp<0.05.Asterisksindicatefewerthan10usersofaresource.KaplanQ-bank,Firecracker,andUSMLERxarequestionbanks.NBMEandUWorldSelfAssessmentsarepracticeexams.DIT,SketchyMicro,andSketchyPharmarevideo-basedresources.RapidReviewandBRSaretexts.
Figure17.AverageStep1scorebrokendownbynumberofQ-bankquestionsanswered.ThefullUWorldQbank,themostcommonlyusedresource,isroughly2350questions.1700thenindicateslessthan50%oftheQ-bankand4000indicates150%.
-8
-6
-4
-2
0
2
4
6
8
KaplanQBank*
NBM
EComprehensive
BasicScienceSelf
UWorldSelf
Assessments
Doctorsin
Training*
Flashcard
programs/Anki
Firecracker
RapidReview
Pathologyby
Goljan
SketchyM
icro
SketchyPharm
BRSPhysiology
USMLERx
AverageScoreDifference
ScoresofUsersvsNon-UsersofResources
190200210220230240250260270
<1700 1701-4000 4001+
AverageStep1Score
#ofQuestions
StepScorevs#ofQ-BankQuestionsAnswered
43
Figure18.AverageStep1scoresbasedonhowmuchpriortotakingtheexamthestudentbegan“intense”studyforStep1.6monthscorrespondedtowinterbreak,and3monthscorrespondedtospringbreak.6weekspriorwasthedefault,asthatwashowmuchtimeeachstudentwasguaranteedfordedicatedstudytime.
Figure19.Averagestepscoresbrokendownbynumberofweeksofdedicatedstudy.
215220225230235240245250255260265270
1yearprior 6monthsprior 3monthsprior 6weeksprior
AverageStep1Score
WhenIntenseStudyBegan
WhentoStartStudyingforStep1
200
210
220
230
240
250
260
4 4.5 5 5.5 6 6.5 7 >7
AverageStep1Score
#WeeksStudying
#WeeksStudyingvsStepScore
44
Figure20.Averagestepscoresbrokendownbytotalhoursstudyingduringthededicatedstudyperiod.
Figure21.AverageStep1scoresbrokendownbylevelofextracurricularinvolvement.
205210215220225230235240245250255
<250 251-300 301-350 351-400 401-450 451-500 501-550 551-600 601+
AverageStep1Score
TotalHoursStudying
TotalHoursStudyingvsStepScore
200210220230240250260270
Signi]icantlylessinvolved(0-1hr)
Somewhatlessinvolved(2-4hr)
Average(4-6hr) Somewhatmoreinvolved(7-9hr)
Signi]icantlymoreinvolved(10+hr)
AverageStep1Score
ExtracurricularInvolvementvsScores
45
Figure22.Averagepredictivevalueofpracticetestscomparedtotheactualexamscore.Theseexamsweremostlytakenwithinaweekoftheactualexam.Errorbarsindicate95%confidenceinterval.
-15
-10
-5
0
5
10
15
Uworld2 NBME13 NBME14 NBME15 NBME16 NBME17 NBME18
AverageScorePrediction
PredictiveValueofPracticeTests(ComparedtoActualUSMLEScore)
46
REFERENCES
1. NationalResidentMatchingProgram.(2016,June).Resultsofthe2016NRMP
ProgramDirectorSurvey.http://www.nrmp.org/.RetrievedFebruary1,2016.
2. London,D.A.,Kwon,R.,Atluru,A.,Maurer,K.,Ben-Ari,R.,&Schaff,P.B.(2016).More
onHowUSMLEStep1ScoresAreChallengingAcademicMedicine.AcademicMedicine,
91(5),609-610.doi:10.1097/acm.0000000000001159
3. NationalResidentMatchingProgram.(2016,June).Resultsofthe2016NRMP
ProgramDirectorSurvey.http://www.nrmp.org/.RetrievedFebruary1,2016.
4. SearchtheAMAResidency&FellowshipDatabase™.(n.d.).RetrievedMarch25,
2017,fromhttps://www.ama-assn.org/life-career/search-ama-residency-
fellowship-database
5. Kleshinski,J.,Khuder,S.A.,Shapiro,J.I.etal.AdvinHealthSciEduc(2009)14:69.
doi:10.1007/s10459-007-9087-x
6. Johnson,J.,Jordan,E.,Burton,W.,&Silhiger,S.(n.d.).AreQuestionstheAnswer?The
EffectofPopularStudyResourcesonUSMLEStep1Performance.Lecture.Retrieved
March10,2017,fromhttps://members.aamc.org/eweb/
7. Prober,CharlesG.MD;Kolars,JosephC.MD;First,LewisR.MD;Melnick,DonaldE.
MD.(2016).APleatoReassesstheRoleofUnitedStatesMedicalLicensing
47
ExaminationStep1ScoresinResidencySelection.AcademicMedicine,91(1),12-15.
doi:10.1097/ACM.0000000000000855
8. Gullo,C.A.,Mccarthy,M.J.,Shapiro,J.I.,&Miller,B.L.(2015).PredictingMedical
StudentSuccessonLicensureExams.MedicalScienceEducator,25(4),447-453.
doi:10.1007/s40670-015-0179-6
9. 34CFR99.31-Underwhatconditionsispriorconsentnotrequiredtodisclose
information?CornellLII.(n.d.).RetrievedMarch23,2017,from
https://www.law.cornell.edu/cfr/text/34/99.31
10. PearsonProduct-MomentCorrelation.(n.d.).RetrievedMarch23,2017,from
https://statistics.laerd.com/statistical-guides/pearson-correlation-coefficient-
statistical-guide.php
11. MedicalSchoolGraduationQuestionnaire2016AllSchoolsSummaryReport.(Rep.).
(2016,June).RetrievedMarch23,2017,fromAAMCwebsite:www.aamc.org/
48
VITAE
SachinShah(July301991-present)isafourthyearmedicalstudentatUTSouthwestern
MedicalSchoolandwillbestartinghispediatricresidencyatUTSouthwesterninJune.He
grewupinPlano,Texas,andwasaEugeneMcDermottScholarasanundergraduate
studentattheUniversityofTexasatDallas.HegraduatedwithaB.S.inBiologyin2013and
immediatelybeganmedicalschool.Inmedicalschool,SachinservedasCo-Presidentofhis
medicalschoolclassforallfouryearsandwasdrawntomedicaleducation.Sachinwill
graduatewithhisM.D.andadistinctioninMedicalEducationinJune2017,andhehopesto
pursueacareerasanattendingphysicianatanacademicteachinghospital.
PermanentAddress:
8024GrandCanyonDr.
Plano,TX75025
49
APPENDIX
APPENDIXA–WEEKLYSTUDENTACADEMICSUPPORTSERVICESSURVEY
APPENDIXB–POST-EXAM1STUDENTAFFAIRSSURVEY
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 1 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…ngPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
Thank you so much for participating in this survey. Your participation will really help futurestudents. We are asking for your name in order to link these responses to other data includingthe weekly surveys you filled out during the dedicated study period. Per IRB protocol, onlyCarol Wortham will be able to link a person to a survey response and will remove all identifiersbefore providing the research team with a completely de-identified data set. If you prefer not tohave your survey responses linked to other de-identified information, you can skip questionone. You also don’t have to answer any question with which you feel uncomfortable.
Thanks again for your time!
1. Step 1 Preparation
Step 1 Preparation 2016
Exit this survey
1. Please select your name from the drop-down list.
2. Which USMLE Step 1 materials/courses did you use?
First Aid for Step 1
USMLE World
Kaplan Q Bank
NBME Comprehensive Basic Science Self Assessments (CBSSA)
UWorld Self Assessments
Doctors in Training Course
Pathoma
Picmonic
Flashcard programs/ AnkiPREVIEW & TEST
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 2 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
Firecracker
Rapid Review Pathology by Goljan
SketchyMicro
SketchyPharm
Med Bullets
Online MedEd
BRS Physiology
Osmosis
Other (please specify)
3. How did you decide what resources to use?
4. How many Kaplan Qbank or USMLE World questions did you complete? (UWorld has2350 questions. You can estimate if unsure.)
5. When in the MS2 year did you start serious (weekly) review/study for Step 1?
1st Semester
January
Spring Break
After Second Year
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 3 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
6. How many weeks did you study full-time for this exam, after MS2 classes ended?
7. When studying full-time, on average how many days per week did you study?
8. When studying full-time, on average how many hours per day did you study?
9. How many passes through First Aid did you make? (Enter 0 if you did not use.)
Why, or why not?
10. Did you find the NBME Comprehensive Basic Science Self Assessment (CBSSA)given after spring break helpful?
Yes
No
11. How many practice exams did you take in addition to the one the school provided?
12. What was the FINAL practice exam that you took? (NBME #, UWorld SelfAssessment #)
13. Did your practice test over- or underestimate your score?
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 4 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
By how many points?
Underestimated
Accurate estimate
Overestimated
14. How many days before your actual test date did you take your last practice exam?
If yes, do you regret this decision?
15. Did you end up moving your USMLE Step 1 exam date?
Yes, I moved it earlier.
Yes, I moved it later by a few days.
Yes, I moved it later by a week or so.
Yes, I moved it later by a month.
No
16. How much money did you spend on Step 1 preparation materials/courses? (notincluding exam administration cost)
17. If finances influenced your choice of materials, please comment.
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 5 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
Please add comments if your agreement or disagreement has a caveat.
18. Please indicate your level of agreement with the following:
I felt very prepared for the USMLE Step 1 based on the curriculum at UT Southwestern.
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
Comments
19. Please indicate your level of agreement with the following:
UT Southwestern provided me with enough dedicated study time to prepare for USMLEStep 1.
Strongly Agree
Agree
Undecided
Disagree
Strongly Disagree
20. Compared to your peers, how involved are you in extracurriculars, volunteering,and/or research?
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 6 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
Significantly less involved
Somewhat less involved
Average
Somewhat more involved
Significantly more involved
21. On an average school week, how many hours did you spend on extracurriculars,volunteering, and/or research?
22. Performance on USMLE Step 1 would be improved by placing a larger emphasis onthe following topics in the MS1 MS2 curriculum. (Please list specific things)
23. My best piece of GENERAL advice for Step 1 preparation is:
StronglyAgree Agree Undecided Disagree
StronglyDisagree Not Utilized
Results of thisSurvey from thePrior Class
24. Please rate your level of agreement with the following statement as it applies to eachservice you used.
I found this service beneficial to my preparation for Step 1. (SASS= Student Academic Support Services)
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 7 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…ngPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
AOA Step-up toStep 1 reviewsessions
AOA/SASS PanelDiscussion on Step1 Preparation
Library SimulatedPractice Exams
SASS Step 1Resource Fair andPacket
SASS CalendarClinic
Personal Consultsin SASS
SASS Step 1Preparation Course
SASS reaching outto me during thepreparation period
Comments
StronglyAgree Agree Undecided Disagree
StronglyDisagree
NotApplicable
First Aid for Step 1
USMLE World
25. Please rate your level of agreement with the following statement as it applies to eachresource you used.
I found this resource beneficial to my preparation for Step 1.
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 8 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
Kaplan Qbank
NBME CBSSA
U World SelfAssessments
Doctors in Trainingcourse
Pathoma
Picmonic
Flashcardprograms/ Anki
Firecracker
Rapid ReviewPathology byGoljan
Sketchy Micro
Sketchy Pharm
Med Bullets
Online MedEd
BRS Physiology
Osmosis
26. If applicable, please provide your feedback/comments on First Aid for Step 1.
27. If applicable, please provide your feedback/comments on USMLE World.
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 9 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
28. If applicable, please provide your feedback/comments on Kaplan Qbank.
29. If applicable, please provide your feedback/comments on NBME ComprehensiveBasic Science Self Assessments.
30. If applicable, please provide your feedback/comments on U World SelfAssessments.
31. If applicable, please provide your feedback/comments on the Doctors in Trainingcourse.
32. If applicable, please provide your feedback/comments on Pathoma.
33. If applicable, please provide your feedback/comments on Picmonic.
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 10 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
34. If applicable, please provide your feedback/comments on Flashcard programs/ Anki
35. If applicable, please provide your feedback/comments on Firecracker.
36. If applicable, please provide your feedback/comments on Rapid Review Pathologyby Goljan.
37. If applicable, please provide your feedback/comments on SketchyMicro.
38. If applicable, please provide your feedback/comments on SketchyPharm.
39. If applicable, please provide your feedback/comments on Med Bullets.
5/2/16, 9:54 PM[SURVEY PREVIEW MODE] Step 1 Preparation 2016 Survey
Page 11 of 12https://www.surveymonkey.com/r/Preview/?sm=WX8nu_2FGtw2v2ITfa…gPSRUCRFUZYyi1O4jb4DqlthKzf_2B5ndVsIVd3jve_2FT21aCo0sNUYHJ0C
40. If applicable, please provide your feedback/comments on Online MedEd.
41. If applicable, please provide your feedback/comments on Osmosis.
42. If applicable, please provide your comments on other resources you used to preparefor Step 1.
43. Before starting the dedicated study period, what was your initial goal for Step 1.
44. In order to evaluate effectiveness of preparation strategies to benefit futurestudents, please indicate your actual score on Step 1.
Thank you very much for your time. The MS2s will be quite grateful as are we.
APPENDIXC–RULESFORCALCULATINGSOCIOECONOMICCATEGORY
SOCIOECONOMICSTATUSQUESTIONS&POINTVALUES GROUPDESIGNATIONSWITHRANGEOFSCORES:Maximumpointsavailabletobeassigned=85
GroupA=45-85totalpointsassignedGroupB=30-40totalpointsassigned GroupC=20-25totalpointsassignedGroupD=0-15totalpointsassigned PARENT’SEDUCATIONALLEVEL: Onlyonesetofpoints,15maximum,willbeincludedintheSEScalculationforparent’seducationallevel.Thepointswillbeassignedtotheparentorsignificantparentalrolepersonwiththelowestlevelofeducationalattainment.Thesignificantparentalrolepersonisconsideredonlyiftheapplicantlivedwiththispersonatleast3yearswhileattendinghighschool.Ifapplicantlivedwiththesignificantparentalpersonforatleast3yearswhileinhighschool,thatpersoniseligibletobeconsideredinassigningtheParents’EducationalLevelpointsintheSEScalculation.Applicantmustanswerinthefollowingway:SignificantMale:
• Didsomeoneotherthanyourbiologicalfatherplayasignificantmaleparentalroleinyourlife?YES
• DidyoulivewiththispersonforatleastthreeyearswhileattendingHighSchool?YES• Ifyes,pleaseprovideeducationlevel.
SignificantFemale:
• Didsomeoneotherthanyourbiologicalmotherplayasignificantfemaleparentalroleinyourlife?YES
• DidyoulivewiththispersonforatleastthreeyearswhileattendingHighSchool?YES• Ifyes,pleaseprovideeducationlevel.
Theprogramwillcheckeducationalleveloffather,motherandeacheligiblesignificantparentalpersonandassignonlyonesetofSESpointsbasedonthelowestlevelofeducationalattainment.
ParentsEducationLevel:
PointsAssigned Choices15 Non-H.S.Graduate10 HighSchoolGraduate/GED5 SomeCollege5 Associate’sDegree0 BachelorDegree0 CollegeBeyondBachelorDegree0 Master’sDegree0 DoctoralDegree0 PostDoctoralStudies
APPLICANT’SHOMETOWNAREA:Howwouldyoudescribetheareawhereyougrewup?
PointsAssigned Choices10 Rural0 Urban10 InnerCity0 Suburban0 MilitaryorGovernmentInstallation0 Other
HOUSEHOLDINWHICHAPPLICANTWASRAISEDORSPENTMAJORITYOFLIFEFROMBIRTHTOAGE18:SizeofHousehold:
PointsAssigned Choices
0 4orLess10 515 620 7orMore
EstimatedValueofResidentialProperty(OwnedorRented):
PointsAssigned Choices15 0-$50,00010 $50,001–$75,0005 $75,001-$100,0000 $100,001-$150,0000 $150,001-$200,0000 Over$200,000
ADDITIONALSESQUESTIONS:
1. Areyouamemberofthefirstgenerationinyourfamilytoapplyto,attendorgraduatefromanundergraduateprogram?a. Yes=10pointsb. No=0points
2. Wereyourequiredtocontributetotheoverallfamilyincome(asopposedtoworkingprimarilyforyourowndiscretionaryspendingmoney)whileattendingelementaryand/orhighschool?a. Yes=15pointsb. No=0points