AnIntegra*ve,Theory-Based,Mul*-FacetedParadigmforSTEMProgramEvalua*on:

LessonsfromtheMeyerhoffScholarsProgram

KennethI.Maton

DepartmentofPsychology,UMBC

PresentedattheUniversityofMichigan,AnnArbor,MI

November29,2016

MeyerhoffResearch:Integra*ve,Theory-Based,Mul*-faceted,

•  OutcomeResearch

•  Theory-based:MechanismsofProgramInfluence

•  ProgramElements:ProgramTheory&MeasureDevelopment

•  QualitaPveInquiry&FormaPveAssessment

•  DocumenPngSystemicandRadiaPngInfluence

MeyerhoffProgramComponents

1.   FinancialAid2.   SummerBridge3.   ProgramValues4.   Advising/Counseling5.   On-campus&Summer

ResearchInternships6.   Facultyinvolvement7.   Recruitment

8.   StudyGroups9.   ProgramCommunity10.  Tutoring11.  Administra*ve

Support12.  Mentors13.  Opencommunica*on

withFamilies

OutcomeResearch•  ExaminetheeffectofMSPversusDeclinedComparison

Sample

•  PreviousFindings–  UndergraduateGPA,RetenPon

–  EntryintoSTEMPhDProgram

•  RecentFindings:STEMPhDComplePon

PreviousFindingsComparedtostudentswhodeclinedtheMeyerhoffscholarshipoffer(comparisongroup),AfricanAmericanMeyerhoffsare:

ü 2#mesmorelikelytograduatewithaSTEMbachelor’sdegree

ü 5#mesmorelikelytopursueaSTEMPhD

STEMPhDComplePonStudy:ParPcipants

***

**

STEMPhDComplePon:Findings

AfricanAmericanMeyerhoffswere4.8*mesmorelikelytocompleteaSTEMPhDthancomparisongroup.Furthermore,oddsraPoanalysesshowthattheyareover7*mesmorelikelytocompleteaSTEMPhD.Also,AfricanAmericanMeyerhoffsfrommorerecentcohorts(1997-2004)weremorelikelytocompleteaSTEMPhD(36.5%)thanthosefromearliercohorts(1989-1996;21.9%).

TopU.S.baccalaureate-originins*tu*onsof2005-14BlackS&Edoctoraterecipients,byins*tu*onalcontrol,2010Carnegie

classifica*on,andHBCUstatus

CarnegieClassificaPonofUniversiPesAfAmMeyerhoffsReceivedPhDDegreesfrom

Theory-Based:MechanismsofProgramInfluence

• PreviousFindings

–  ResearchExcitement(atcollegeentry)–  ResearchInvolvement(duringcollege)–  ProgramComponents(meanraPngs)

• RecentStudy: DoesperceivedprogrambenefitmediatetherelaPon betweensenseofcommunityandscienceidenPty& researchself-efficacy?”

ParPcipants•  Meyerhoffs(N=109)enteringbetween2010&2013

•  Gender:57.8%Male;&42.2%female

•  Race/Ethnicity:55%URM

•  MeanSATscores:Math=696.2,Verbal=628.4,WriPng=632.9

•  MeanweightedhighschoolGPA=4.1

Measures*•  SenseofCommunityinMSP.Collectedatendof6-weekSummerBridge.

Measuredby12-itemsadaptedfromtheSenseofCommunityIndex.

–  Sampleitems:Icantrustpeopleintheprogram.”;“IexpecttobeapartoftheprogramforalongPme.”

•  PerceivedProgramBenefit.Collectedatendofyear1.Measuredby17-itemsthatassess”howuseful”eachprogramelementistostudents.

–  Financialaid,studygroups,tutoring,academicadvising,etc.*Scalescores=meanofallitems.Measureshave“strong”reliabilityesPmates.

Measures*Scienceiden#ty.Measuredby5-itemsfromScien*ficIden*tyScalethatassesshowmuchbeingascienPstisapartofstudentidenPty(Chemers,etal.,2010)

–  Sampleitems:“IhaveastrongsenseofbelongingtothecommunityofscienPsts”;“IfeellikeIbelonginthefieldofscience”

ResearchSelf-Efficacy.Measuredby6-itemsfromScien*ficSelf-EfficacyScalethatassessstudents’abilitytofuncPonasascienPst.

–  Sampleitems:“GeneratearesearchquesPon”;“UsescienPficliteratureand/orreportstoguideresearchanddeveloptheories.”

*Collectedatendofyear2.Scalescores=meanofallitems.Measureshave“strong”reliability.

Findings:ScienceIdenPty

PerceivedProgramBenefit

(endoffreshmanyear)

ScienceIden*ty(endofsophomoreyear)

PsychologicalSenseofCommunity

(endofSummerBridge)

.38**.31**

.12(.24**)

Note:**p<.01

Findings:ResearchSelf-Efficacy

PerceivedProgramBenefit

(endoffreshmanyear)

ResearchSelf-Efficacy

(endofsophomoreyear)

PsychologicalSenseofCommunity

(endofSummerBridge)

.38**.33**

.13(.26**)

Note:**p<.01

ProgramElements:ProgramTheoryandMeasurementDevelopment

•  PreviousResearch

–  TheoryDevelopment–  MeanRaPngsofElements

•  RecentResearch:MeasureDevelopment

ProgramElements:MeasureDevelopment

•  SetofinterviewswithProgramDirectorandAsst.

Director–  MeyerhoffModel(programtheory)–  IntendedOutcomes

•  DevelopmentofPilotMeasure–  GeneraPonofiniPalsetofitems–  Itemreview,criPqueandrefinement–  PilottesPngwithsampleofstudents–  PsychometricAnalyses

•  InclusioninEndofAcademicYearSurvey

ProgramValuesElement:Focus

•  Programinculcatesinstudentsacommitmentto– Excellence– Accountability– Groupsuccess– Community– Leadership– Professionalism– Characterandintegrity– DiversityinSTEM– PursuitofaSTEMPhD

ProgramValuesElement:IntendedOutcomes

§  FormaPonofMeyerhoffidenPty§  UnifiedunderstandingofprogramintenPons§  InternalizaPonofvaluesandvaluepursuitthroughoutcareer§  AppliedpassionfordiversityinSTEMfields§  Accountabilityamongscholars§  AcPveparPcipaPoninbuildingsupportstructuresforscholars,

throughpeersupport§  StrengthenedcommitmenttoPhDpursuits

SenseofCommunityElement:Focus

•  Family-likesocialandacademicsupportsystem•  Liveinthesameresidencehallfirstyear•  Requiredtoliveoncampussubsequentyears.•  Staffregularlyhold“familymeePngs”•  IntenPonalcreaPonofacommunityofscholars–  “MeyerhoffFamily”concept–  Livingandlearningcommunity–  RegularGroupmeePngs–  ResidenPalrequirement–  Parentalinvolvement– MeyerhoffStudentCouncil

SenseofCommunityElement:IntendedOutcomes

•  Supportstructureforstudentsoflike-mindedpeers•  Enhancedsenseofcommunity(cohort;program)•  Avenuesforleadershipandinfluence•  AppreciaPon&pracPceworking/succeedingtogether•  EnhancedacademicandsocialintegraPon•  CommitmenttoSTEMcareerthroughcommunitydiscourse•  Increasedsocialandculturalnetworks

Scales:ReliabilitiesandMeans

Scale(#Items) Reliability Mean S.D. N Recruitment/SelecPonWeekend(5) .79 4.2 .63 112SummerBridge(7) .81 4.2 .60 110Financialsupport(6) .85 4.1 .73 112ProgramValues(9) .83 4.4 .49 117Advising(10) .92 4.2.69 118Researchexperience(7) .83 4.2 .62 93AdministratorSupport(4) .79 4.4 .54 118ProgramCommunity(14) .92 4.2 .59 118 StudyGroups(7) .88 4.4 .55 117FacultyInvolvement(5) .78 4.0 .66 118

Scales:Intercorrelations

Scale Min MaxMeanScale-TotalRecruitment/SelecPonWeekend(5) .24 .53 .41 .59 SummerBridge(7) .22 .69 .45 .62Financialsupport(6) .08 .40 .25 .34ProgramValues(9) .29 .75 .54 .77Advising(10) .26 .70.53 .69StudyGroups(7) .14 .61 .39 .51ProgramCommunity(14) .33 .70 .56 .84Researchexperience(7) .37 .75 .48 .67 FacultyInvolvement(5) .27 .53 .40 .56AdministratorSupport(4) .08 .55 .43 .61

QualitativeInquiry&FormativeAssessment

•  Interviews•  FocusGroups•  ObservaPon•  Surveys

QualitativeInquiry:SelectionWeekend

“TheyspendalotofBmeplanning[SelecBonWeekend]andittakesalotofpeopleandalotofmanhourstodoit.Andwhenyougo,youcantellthatthey’vespentalotofBmeandmoneytobringyouhere...andthey’respendingthatmuchBmeonyou.Youknow,thattellsyousomethingabouthowtheyseethestudentsintheprogram.”

“SummerBridgeiswhereitallbegins.It’slikethesayingthatachainisonlyasstrongasitsweakestlink.That’sSummerBridge.”

QualitativeInquiry:SummerBridge

QualitaPveInquiry:StaffandPeerAdvising/Counseling

“Theguidancethattheygive[ismosthelpful]….ifIwerecomingtoUMBCandIwasn’taMeyerhoffscholar,I’dhavenoideawheretogo.IknowIwanttomajorinBiochemistrybutI’mnotsurewhat’sthebestpathtodothat,whatarethebestclassestotake,whatarethebestinternshipstodo….theMeyerhoffprogramwillhelpdothat.”

Qualita*veInquiryPROGRAMCOMMUNITY:FamilyMee*ng

“...Youcouldtell[Dr.Hrabowski]andallthestaff...justgenuinelywantedtoseeyousucceed.Thattheyweregoingtotrytodoeverything,aslongasyoudidyourbest,theywoulddotheirbesttojusthelpyouanywaytheycould.Andthatwastheirmaingoal.”

QualitaPveInquiry:ResearchExperience/PresentaPons

“TheMeyerhoffProgramactuallygavemeanopportunitytodoresearch....Itwasmyfirstbenchscienceexperience....AndIjustfellinlovewithit.IjustreallylovedhowIhadtothinkabouteverythinganditjustwasverynewforme.AndthathadalottodowithconsideringthePhD.EverysummerIdidanewinternshipandIlearnedveryquicklythatinternshipsgetyouaroundthecountry.[chuckling]SoIhadajoyfulkindofthoughtaboutresearch,justaZachedtoresearch...SoIjusthadtheserecurrentreinforcementsaboutresearchandlife.”

Documen*ngSystemicandRadia*ngImpact

•  PriorScholarship–  Inclusiveexcellence,changingcampuscultureandclimate(theoryof

insPtuPonalculturechange)–  ScienceArPcle(undergraduateimpact)

•  RecentScholarship–  UndergraduateImpact(updated)–  SynergisPcEffectonProgramDevelopment–  BeyondUMBC–  MeyerhoffAdaptaPonProgram(HHMI)

Campus-WideImpact:Undergraduate

•  NumberofAfrican-AmericanUndergraduatesmajoringinscienceandengineeringincreasedmorethat11.5-foldsince1985

•  OverallandS&EenrollmentsamongLa*nostudentsgrew6.1and16.1-fold,respec*vely,since1985

•  NumberofCaucasianundergraduatesmajoringinS&Eincreasedmorethan3.1-foldsince1985

•  Avg.GPAofAfrican-AmericanstudentsmajoringinS&Eincreasedfrom2.7in1989to3.2in2014

1988 20002014

2000:CWITScholars

1988:HonorsCollege1988:MeyerhoffProgram

1997:MARC

1992:HumaniPesScholars1994:HHMIScholars

1995:LinehanAr*stScholars

1999:SondheimPublicAffairsScholars2000:LivingLearningCommuniPes

2004:IHU

2008:CollegiateSuccessInsPtute

2002:FirstYearSeminars

2005:ChemistryDiscoveryCenter

2010:CASTLE-Courseredesign2010:TRS

2008:LearningResourcesCenter

2010:I-cubed@UMBC

2003:ADVANCE

2008:LSAMP

2011:TransferSuccess

2004:PhDComplePonProject

2002:AGEPPromise

2012:InnovaPonFund

2014:STEMBUILD

Meyerhoff model -

Promoting broader institutional change across majors from BA through PhD Evaluation as integral

BeyondUMBC:MeyerhoffAdapta*ons

•  SchoolsthathaveusedtheMeyerhoffScholarsProgramasamodelforprogramsontheircampusesinclude:–  BatesCollege–  CornellUniversity– DukeUniversity–  LouisianaStateUniversity– MorehouseCollege– UniversityofMichigan– Winston-SalemUniversity– WinthropUniversity

MeyerhoffAdapta*onPartnershipProject(MAP)

•  AlliancebetweenHHMI&– UNC-ChapelHill–  PennState– UMBC

•  Ra*onale–  ToincreaseURMrepresenta*oninSTEM

MAPAssessment:IntegratedResearchDesign

Mul*-LevelImplementa*onAssessment

YearlyStudentAcademicOutcomes

ProgramComponentImplementa*on

Value-AddedStudentAcademicOutcomes

Ins*tu*onalImpactonScienceCultureandURMSuccess.Forma*veAssessmentandFeedbackModelingAnalysesPartnershipAssessmentConsulta*onIntegra*veReportofFindings

LessonsLearned

Outcome/SummaPveEvaluaPon

•  Mul*pleComparisonSamples–  UMBC,Declined,Na*onal(NSF)

•  Mul*pleOutcomeMeasures–  GPA,STEMMajor,PhDEntry,PhDReceipt

•  Longitudinal–  Trackingover*me,Mul*plecohorts,Funding

•  Covariates–  Gender,Race/Ethnicity,SAT,HSGPA,YearofEntry

Theory-Based/Mechanisms

•  Genera*ngProgramTheory–  Mul*plesources(literature,program)Itera*ve

•  MixedMethodsDesign(convergence)–  Surveys,Interviews,Focusgroups,Observa*on

•  Measurement–  Exis*ngmeasures,Newlydevelopedmeasures

•  Mul*pleCollaborators,VariedBackgrounds

–  Applied,Theory-based,Mul*pledisciplines/perspec*ves

FormaPve/Process-Focused

•  Importance–  ProgramDevelopmentandImprovement

•  Mul*pleMethods–  Observa*on,Interviews,FocusGroups,Surveys

•  Collabora*veProcess–  Protocoldevelopment,*ming,dissemina*on

•  MeansofSharing(formal&informal)

–  Reports,Presenta*ons,Lunch,Hallways

PeopleandRelaPonships

•  Trus*ngRela*onships(UMBC-wide)•  Reciprocity•  Flexibility•  InterpersonalSkills•  ResearchTeam(studentsofcolor)

Acknowledgements•  CurrentFundingSources:HowardHughesMedicalInsPtute,

NaPonalInstutesofHealth,NaPonalScienceFoundaPon

•  ResearchTeamMembers:LabResearchScienPst,Post-DoctoralResearchAssociate,GraduateandUndergraduateStudents

•  HHMIAssessmentTeamsatPennStateandUNC-ChapelHill

References(Selected)Maton,K.I.,Beason,T.S.,Godsay,S.,Sto.Domingo,M.R.,Bailey,T.C.,Shun,S.,&HrabowskiIII,F.A.(2016).OutcomesandprocessesintheMeyerhoffScholarsProgram:STEMPhDcomplePon,senseofcommunity,perceivedprogrambenefit,andresearchself-efficacy.CBE-LifeSciencesEducaBon,15(3).Maton,K.I.Pollard,S.A.,Weise,T.V.M.,&Hrabowski,F.A.III.(2012).TheMeyerhoffScholarsProgram:Astrengths-based,insPtuPon-wideapproachtoincreasingdiversityinSTEM.Mt.SinaiJournalofMedicine,79,610-623.

Maton,K.I.,Hrabowski,F.A.,Ozdemir,M.&Wimms,H.(2008).EnhancingrepresentaPon,retenPonandachievementofminoritystudentsinhighereducaPon:AsocialtransformaPontheoryofchange.InM.Shinn,&H.Yoshikawa,H.(Eds.),TowardposiBveyouthdevelopment:Transformingschoolsandcommunityprograms(pp.115-132).NewYork:OxfordUniversityPress.