aosl research report - the impact of arts-based … · the impact of arts-based innovation training...
TRANSCRIPT
A u d i e n c e V i e w p o i n t s C o n s u l t i n g
13148RoundingRunCircle,Herndon,VA20171t:831-224-3085www.audienceviewpoints.com
TheImpactofArts-BasedInnovationTrainingontheCreativeThinkingSkills,CollaborativeBehaviorsandInnovationOutcomesofAdolescentsandAdults
AResearchStudyReport
Preparedby:
KateHaleyGoldmanStevenYalowitz,Ph.D.
ErinWilcox,M.A.
August3,2016
ThisresearchwasfundedbygrantDRL-1224111fromtheNationalScienceFoundation.HarveySeifter,PrincipalInvestigator©ArtofScienceLearning,2016.Allrightsreserved.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 2
TableofContents
TableofContents 2
ExecutiveSummary 4
DiscussionandImplicationsforFutureResearch 11
IntroductionandProjectBackground 14
StudyDesign 16
Methods 20RecruitmentSurvey 22PreWorkshopSurvey 23PreWorkshopCreativeThinkingSkillsTest 23WorkshopObservations 24TeamCollaborationRatings 24PostWorkshopCreativeThinkingSkillsTest 25PostWorkshopSurvey 25TransferabilitySurvey 25TeamInnovationOutcomesAssessmentPanelRatings 25
Limitations 27
SamplingandCharacteristicsoftheSample 29HighSchoolStudents 29EarlyCareerSTEMProfessionals 32
Hypothesis1Findings:CreativeThinkingSkills 37DifferencesWithinGroups 38DifferencesBetweenGroups 39CriticalThinking 46CreativeThinkingSkillsSelf-Report 47
CreativeCompetenciesInventory(mini-ECCI) 47CreativeProblemSolvingProfile(CPSP) 48
TransferabilityofSkills 52DefinitionofInnovation 54
HighSchoolStudents 54EarlyCareerSTEMProfessionals 56
SelfPerceptionasInnovator 58HighSchoolStudents 58EarlyCareerSTEMProfessionals 64
UsingChallengeSkillsintheFuture 74
Hypothesis2Findings:CollaborationBehaviors 77ObservedCollaboration 77Self-ReportedCollaboration 78RelationshipBetweenObservedandSelf-ReportedCollaboration 79HighSchoolStudentObservedBehaviorFindings 79
ComparisonBetweenControlandTreatmentbyBehavior 80
AudienceViewpointsConsultingArtofScienceLearningResearchReport 3
ComparisonBetweenControlandTreatmentintheFinalSession 89Self-ReportedTeamCollaborationRatingsofHighSchoolStudents 91HighSchoolStudentBehaviorFindingsSummary 95EarlyCareerSTEMProfessionalBehaviorFindings 96
ComparisonBetweenControlandTreatmentbyBehavior 96ComparisonBetweenControlandTreatmentintheFinalSession 105
Self-ReportedTeamCollaborationRatingsofEarlyCareerSTEMProfessionals 107EarlyCareerSTEMProfessionalsBehaviorFindingsSummary 111
Hypothesis3Findings:TeamInnovationOutcomes 112HighSchoolStudentsFindings 113EarlyCareerSTEMProfessionalsFindings 114TeamInnovationOutcomesFindingsSummary 116
References 117
Appendices 118AppendixA:PreWorkshopSurvey 118AppendixB:PostWorkshopSurvey 123AppendixC:ObservationRating 127AppendixD:CollaborationRating 128AppendixE:TransferabilitySurvey 129AppendixF:PreWorkshopCreativitySkillsTaskWorcester 132AppendixG:PreWorkshopCreativitySkillsTaskSanDiego 134AppendixH:PostWorkshopCreativitySkillsTask 136AppendixI:CreativitySkillsTasksRubric 138AppendixJ:BusinessCase 141AppendixK:OutputScoringSheet 143AppendixL:Inter-raterReliabilityScoringforCreativitySkillsTest 146AppendixM:IdeaClusterCategoriesforCreativitySkillsTest 147
AudienceViewpointsConsultingArtofScienceLearningResearchReport 4
ExecutiveSummaryPROJECTBACKGROUND:TheArtofScienceLearningProject(AoSL)isaNationalScienceFoundation(NSF)-fundedinitiative,foundedanddirectedbyHarveySeifter,thatusestheartstosparkcreativityinscienceeducationandthedevelopmentofaninnovative21stCenturySTEM(Science,Technology,EngineeringandMath)workforce.In2007,Seifter,alongwithartist/scientistToddSilerandchoreographerLizLerman,ledanNSFsymposiumontherelationshipbetweenthearts,STEMlearningandworkforcedevelopment.In2008,SeifterandcolleaguesatNewYork’sLearningWorldsInstituteheldaseriesofroundtableswithscienceeducators,whichrevealedabroadlysharedbeliefintheconnectionbetweentheinvestigativenatureofscienceandthearts,andanappreciationforthepotentialofarts-basedlearningtofosterpassionforexplorationanddiscoveryinyounglearners.Thesemeetingsplayedanimportantroleindesigningaproposal,whichwassubsequentlyfundedbytheNationalScienceFoundation(DRL-0943769).In2011,Phase1oftheprojectconvened425scienceeducators,teachingartists,museumprofessionals,classroomteachers,businessleaders,policymakers,andacademicresearchersinregionalconferencesattheSmithsonianInstitution,IllinoisInstituteofTechnologyandCaliforniaInstituteofTelecommunicationsandInformationTechnology(Calit2).Thegoalsweretoexploretheconnectionbetweenthearts,innovationandeconomiccompetitiveness;createcommunitiesofpracticebysharingeducationalresources,curricula,andbestpracticesthatuseABLtostrengthenSTEMlearning;andexperiencefirst-handarts-basededucationaltechniquesthatdevelopcriticalandcollaborativethinkersfortheSTEMworkforce.AtthewritingofthisreporttheArtofScienceLearningprojectisinPhase2,fundedbytheNSF(DRL-1224111)todevelopanewarts-basedSTEMinnovationcurriculumforadolescentandadultlearners;threeyear-longarts-basedincubatorsforinnovationinSTEMlearningandpracticetotestandrefinethecurriculum;atravelingart/scienceexhibition;andpublicprogramsthatusetheproject’sactivitiesandoutcomestoadvancecivicengagementwithSTEM.Phase2alsoincludedresearchcomparingtheimpactofarts-infusedSTEMinnovationtrainingwithtraditionalproject-basedSTEMinnovationtraining,amulti-yearresearchprojectthatwasindependentlycarriedoutbyAudienceViewpointsConsulting.ThisreportcontainstheresultsofthisPhase2research. RESEARCHDESIGN:AoSL’sresearchcomponentwasdesignedtotesttheideathatintegratingtheartsintoSTEM-relatedinnovationtrainingwouldresultinenhancedcreativethinkingskills,moreextensivecollaboration,morerobustinnovationprocessesandimprovedinnovationoutcomes.Twocities,Worcester,MassachusettsandSanDiego,California,servedasthesitesfortheresearchstudy.HighschoolstudentswerethesamplepopulationinMassachusetts,andearlycareerSTEMprofessionalsinCalifornia.AtbothsitestheAoSLprojectteamhostedfiveweek-longinnovationtrainingsessions,witheachgroupmeetingforahalfdayperweek,totalingroughly20hoursofinvolvementforeachparticipantintheresearchproject.Thetrainingsessionsinvolvedproject-basedlearningfocusedonthefrontendofinnovation,withprojectsaddressinglocalSTEMchallenges(transportationalternativesinWorcester,waterresourcesinSanDiego).Overthecourseofthefiveweeks,teamsofparticipantscreatedsimpleprototypesandbusinesscasesfornewproducts,processesandservicesintendedtoaddressthesechallenges.Thetrainingcurriculum,groundedinbestpracticesderivedfromtheProductDevelopmentManagementAssociationBodyofKnowledge,includedthekeyconceptsofinnovation,STEMcontentspecifictoeachlocalchallenge,andcollaborativeprojectinnovationactivitiesandexercises.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 5
Therewerethreemainhypothesesthatguidedthisresearch:1. Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,improvesan
individual'screativethinkingskillsincludingcriticalthinking,divergentthinking,problemidentification,convergentthinkingandproblemsolving.
2. Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,increasesindividualcollaborativebehaviorswithinateamcontext.
3. Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,enhancesthenovelty,impactandfeasibilityofteaminnovationoutcomes.
Inordertotestthesehypotheses,theresearchstudyusedaquasi-experimentaldesignwithapre-test,post-testintactgroupdesign,includingacontrolgroupforcomparisonpurposes.Intactgroupdesignmeansthatthesameparticipantsfilledoutthepre-testandpost-test,inordertocomparehowresponseschangedfromthebeginningtotheendoftheirparticipation.Individualswhoparticipatedinthestudyweregivenapre-recruitmentsurveyandbasedonthiswereassignedtoeitherthecontrolortreatmentgroup,andcounter-balancedbasedonrelatedvariablessuchasinterestandexperiencewiththeartsandsciences.Individualsweredistributedasevenlyaspossiblebetweenthecontrolandtreatmentgroupsbasedondemographicandpsychographicvariablescollectedduringthepre-test.METHODS:ArtofScienceLearningIncubatorsatEcoTarium(Worcester)andBalboaParkCulturalPartnership(SanDiego)servedashostsitesfortheresearchstudies,withArtofScienceLearningstaffembeddedinbothinstitutionsprovidinglocaladministrativesupport.Averysimilarrecruitmentapproachwasusedinbothcohorts(WorcesterandSanDiego)wherethelocalteammemberssentaninvitationonbehalfoftheresearcherproject.HighschoolstudentswererecruitedfortheWorcestercohort,andearlycareerSTEMprofessionalswererecruitedfortheSanDiegocohort.Bothgroupswererecruitedbasedonanumberofcriteria(age,experiencewithSTEM,experiencewiththearts,etc.).Atotalof16groupsparticipatedinthestudy:8studentgroups(4control,4treatment)inWorcester,and8earlycareerSTEMprofessionaladultgroups(4control,4treatment)inSanDiego.Eachgroupincluded7to10individuals.Bothcontrolandtreatmenttrainingswereheldatseparatetimes,andwhileparticipantsknewthattherewasanothersimilargroupmeetingthesameday,theywereunawarethatthetrainingvaried.Controlandtreatmentgroupsbothusedhands-onproject-basedlearningandanapproachtoinnovationgroundedinBestPracticesfromtheProductDevelopmentManagementAssociation,asarticulatedinthePDMA2014BodyofKnowledge(Kahn,2013).Thetreatmentcurriculumreplaced9hoursofthetraditionalinnovationpedagogyusedinthecontrolcurriculumwith9hoursofarts-basedactivitiesdesignedtoachievethesamelearningobjectives.Inthismanneronlytheapproachwasvaried,toprovideforthecleanestcomparisonofthetwoapproaches.Thereweresevenmethodsused,inordertotriangulatetheresearchfindings:1)arecruitmentsurveyfromthoseinterestedtodetermineeligibilityforparticipation,2)pre-workshopsurvey,3)post-workshopsurvey,4)creativethinkingskillsassessments,5)observationsofgroupsduringweeklymeetings,6)afollow-uptransferabilityofskillslearnedsurveywithasubsetofparticipants,and7)scoringoftheteam’sinnovationproducts,processesandservicesbyanexpertpanelofjudges.Methods1and2wereconductedbeforethetrainingbegan;method3wasconductedintheweeksfollowingthetraining;4and5includeddatacollectedduringthe5-weektrainingperiod;andmethods6and7wereconductedseveralmonthsfollowingthetraining.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 6
MAINFINDINGS(basedonthehypotheses):Hypothesis1:Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,improvesanindividual'screativethinkingskillsincludingcriticalthinking,divergentthinking,problemidentification,convergentthinkingandproblemsolving.MappingChangesinCreativeThinkingAscreativityisacomplexconstruct,thereweremultiplemeasurementsofdifferentattributesofcreativitywithintheresearch,includingvalidatedscalesfromotherstudiesaswellasinstrumentsanditemscreatedforthisproject.Scalesfromotherstudiesaskedparticipantstoratetheirpersonalcapacitiestowardsavarietyofcreativeprocesses,andtheirpreferencesforcreativityatavarietyofstagesofdevelopment.TheseincludedtheECCIscale(Epstein,Schmidt,&Warfel,2008),twocriticalthinkingscalesandtheCreativeProblemSolvingProfile(CPSP)developedbyBasadur,Graen,andWakabayashi(1990),whichmeasuresindividualstrengthwithinfourdifferentcomponentsofthecreativityprocess:generation,conceptualization,optimization,andimplementation.AcreativityskillsinstrumentcreatedforthisprojectaskedparticipantstoidentifyproblemsrelatedtoagivenInnovationChallenge,selectonetoworkon,generatepossiblesolutionstotheselectedproblem,selectonesolution,andexplaintheirchoices(seeAppendicesFthroughHforthefullexercises).Thesameexercisewasgiventwice:onceintheopening15minutesoftheinitialsession,andagainduringtheclosing15minutesofthefinalsessionfiveweekslater.AdifferentInnovationChallengewasusedfortheseconduseoftheexercisetopreventanypracticeeffects.Aproject-developedparticipanttransferabilityofskillssurveymeasuredtheextenttowhichengaginginthefive-weekresearchchallengehadresidualimpactfourmonthslater.Theresearchteamwasinterestedinwhetherparticipantswereabletoapplywhattheydidduringthetrainingtotheirownsubsequentexperiences,includingschool,extracurricularactivitiesandhomeorpersonallives,andtowhatextentparticipantsexpectedthatimpacttocontinueorgrowinthefuture.HighSchoolStudentsOverall,creativecompetencies,asmeasuredbytheECCIscale,significantlyincreasedinthehighschooltreatmentgroup,anddecreased(thoughnotsignificantly)withinthecontrolgroup.TherewerenosignificantdifferencesbetweentreatmentandcontrolgroupchangescoresforhighschoolstudentsontheCPSPscale.Someofthemoststrikingfindingswerewithinthemetricsfromtheoutcomesofthecreativityskillstest–7ofthe16creativeskillsmeasuresshowedastatisticallysignificantincreasefromthepre-testtothepost-testforthetreatmentgroup.Thesedifferenceswerewithinbothconvergentanddivergentskills,withstrongerevidenceforanincreaseindivergentthinkingskills.Forthemeasuresspecificallyaboutcriticalthinking,statisticallysignificantdifferenceswerefoundbetweenthecontrolandtreatmentgroupsforbothofthemaincriticalthinkingscalesused,withthetreatmentgroupscoringasmuchasthree-quartersofapointhigher.Inthiscase,therewerenodifferencesbetweenpre-andpost-testmeasuresforthetreatmentgroup.Thestudentswhowereinthecontrolgroupscoredsignificantlyloweronthepost-testcomparedtothepre-test.Thecombinationoflackofchangeinthetreatmentgroupandadecreaseinscoreswithinthecontrolgroupresultedinstatisticallysignificantdifferencesbetweenthegroups,butnoevidenceofgainwithineitherofthehighschoolgroups.Significantdifferencesalsoemergedaroundskilltransferability.Highschoolstudentsinthetreatmentgroupweremorelikelytoreportapositiveimpactandanticipatefutureimpactfromtheseexperiences(comparedtothecontrolgroup),withsomeofthedifferencesbeingquitelarge.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 7
EarlyCareerSTEMProfessionalsTherewasastatisticallysignificantdecreaseincreativityinbothtreatmentandcontrolgroupsinearlycareerSTEMprofessionalsontheECCIscale.TherewerenosignificantdifferencesbetweentreatmentandcontrolgroupchangescoresforearlycareerSTEMprofessionalsontheCPSPscale.BoththeearlycareerSTEMprofessionalstreatmentandthecontrolgroupsshowedsomeincreasesincreativeskillsfrompretesttoposttest,includingwithinmeasuresofconvergentanddivergentthinking.Thetwogroupsshowedgainswithindifferentskills.Whencomparedagainstoneanother,onestatisticallysignificantdifferencebetweenthetreatmentandcontrolgroupsemerged,intheabilitytoidentifyandclearlyframeproblemsarisingfromagivenchallenge.NodifferenceswerefoundbetweentheearlycareerSTEMprofessionalsgroupsinthecriticalthinkingscale.Thetreatmentandthecontrolgroupsbothhadslightincreasesfromthepretoposttests,andnodifferenceswerefoundbetweenthetwogroups.Therangeofscoreswaslarge,meaningthatboththearts-basedandthetraditionalinnovationtraininghaddifferentiatingeffectsoncriticalthinkingforadults,inthatsomebenefitedgreatlybutotherslostground.Thereislikelysomeothervariableorsetofvariablesthatdetermineshowthetrainingwillimpactcriticalthinkingskills;however,analysisofthevariablestodatehasnotuncoveredspecificleadsonwhatthoseinfluencesmightbe.Nosignificantdifferencesbetweencontrolandtreatmentwerefoundaroundskilltransferability.Hypothesis2:Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,increasesindividualcollaborativebehaviorswithinateamcontext.AssessingIndividualCollaborativeBehaviorResearchersobservedthebehaviorsofindividualswithintheirgroupsduringsubstantialpartsofeachofthesessionstheywereworkingtogetheroverthefive-weekperiod,trackingchangesintheprevalenceofspecificbehaviorsofindividualsineachgroupovertime.Inanattempttotriangulatearealisticdepictionofanindividual’scollaborationandparticipationintheInnovationChallenge,attheendofeachworkshopsessionparticipantswerealsoaskedtoratethemselves,andeachindividualontheirteams,onaseriesofbehaviorsthatalignedwiththebehaviorsrecordedbydatacollectorsintheworkshopobservationsheet.HighSchoolStudentsBasedonobservationaldata,comparisonsweremadeoneachbehavioroverthefive-weekperiodforboththecontrolandtreatmentgroups.Inlookingateachgroupindividually,bothtreatmentandcontrolgroupsshowedsimilar,andstatisticallysignificant,increasesintrustinmovingtowardsasolution,beingtransparentincommunication,theabilitytodisagreeproductively,creatingacultureofmutualresponsibilityandproductivelymanagingdisruption.Controlgroupsshowedstatisticallysignificantincreasesoverthefive-weekperiodinsharingleadership,beingtransparentincommunication,definingacommonpurpose,andcreatingacultureofmutualaccountability.Treatmentgroupsshowedstatisticallysignificantincreasesoverthefive-weekperiodinemotionallyintelligentbehavior,empathiclistening,andtheabilitytodisagreeproductively.Whencomparingthetwogroupsdirectly,therewere8ofthe11behaviorswherethefrequencyandpatternsofthebehaviorsdifferedsignificantlybetweencontrolandtreatment.In6ofthese8behaviors,thetreatmentgroupshowedthestrongerperformance:sharesleadership,trustinmovingtowardasolution,transparentincommunication,emotionallyintelligentbehavior,disagreeproductivelyanddefiningacommonpurpose.Intheremaining2ofthese8behaviors(creatingacultureofmutualaccountabilityandproductivelymanagesdisruption)thecontrolgroupshowedmarginally,
AudienceViewpointsConsultingArtofScienceLearningResearchReport 8
butstatisticallysignificant,strongerperformance.Anadditionalanalysisallowedforacomparisonbetweenthecontrolandtreatmentgroupsduringtheirfinal(R5)sessionswhenparticipantswerecompletingtheircourseofstudyandteamswerefinishingtheirworkonthechallengeandmakingalloftheirfinaldecisionswithrespecttobusinesscases(seeAppendixJ)andpresentations.Thus,R5datagaveasenseofthecumulativeimpactofthefulltwenty-hourinterventiononcollaborativebehaviorofcontrolandtreatmentgroups.Whencomparingthetwogroupsdirectly,statisticallysignificantdifferenceswereseenwithrespecttothefrequencyoffivebehaviorsduringthissession:sharesleadership,emotionallyintelligentbehaviors,mutualrespect,abilitytodisagreeproductively,anddefiningacommonpurpose.Alloftheseshowedahigherlevelofoccurrenceforthetreatmentgroup.Self-reportedteamcollaborationratingsweremarkedlydifferent;onlytwoitems(mutualrespectandtrust)showedasignificantchangefrompre-testtopost-test,anditwasthecontrolgroup,ratherthanthetreatmentgroup,thatshowedasignificantincreaseinbothcases.Treatmentgroupsshowedstatisticallysignificantgreaterincreasesinsharingleadership,emotionallyintelligentbehavioranddefiningacommonpurpose,comparedwiththecontrolgroups.EarlyCareerSTEMProfessionalsTheobservationaldatarevealsignificantpre/postincreasesinsevencollaborativebehaviorsamongthetreatmentgroups:sharingleadership,activefollowing,emotionallyintelligentbehavior,empathiclistening,mutualrespect,trustinmovingtowardsasolution,andtransparencyincommunication.Onlyoneofthesebehaviors(emotionallyintelligentbehavior)alsosawanincreaseamongthecontrolgroupoverthefiveweeks.Whencomparingthetwogroupsdirectly,therewere7ofthe11behaviorswherethefrequencyandpatternsofthebehaviorsdifferedsignificantlybetweencontrolandtreatment.In4ofthese7behaviors,thetreatmentgroupshowedtheunambiguouslystrongerperformance.Thesebehaviorswereactivefollower,mutualrespect,trustinmovingtowardasolution,andtransparentincommunication.In2ofthebehaviors,sharingleadershipandempathiclistening,thecontrolgroupshowedamarginally,butstatisticallysignificant,strongerperformance.Inemotionallyintelligentbehavior,thetreatmentgroupshowedamarginally,butstatisticallysignificant,strongerperformance.Incomparingjustthelastsession,therewerestatisticallysignificantbehavioraldifferencesfor2ofthe11observedbehaviorswereobserved:mutualrespectandtrustinmovingtowardsasolution.Forbothofthese,thetreatmentgrouphadasignificantlyhigheroccurrenceofthesebehaviors. Onceagain,therewerestrikingdifferencesbetweenobservationaldataandself-report;therewerenostatisticallysignificantdifferencesfortheself-reportedteamcollaborationmeasuresfortheearlycareerSTEMprofessionals.Hypothesis3:Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,enhancesthenovelty,impactandfeasibilityofteaminnovationoutcomesJudgingInnovationOutcomesApanelofthreenationalexperts,drawnfromtheselectioncommitteeoftheProductDevelopmentManagementAssociation(PDMA)’sOutstandingCorporateInnovatorAwards,developedanassessment
AudienceViewpointsConsultingArtofScienceLearningResearchReport 9
rubricidentifyingandweightingsevenmeasurestogaugethequalityoftheinnovationoutputs,andsubsequentlyappliedtherubrictothenewproduct,processandserviceconceptsdevelopedbytheteams.PanelistsassembledattheUniversityofIndiana’sKellySchoolofBusinesstoreviewbusinesscasescreatedbytheteams(workingonatemplatedevelopedbyHarveySeifter),PowerPointpresentationscreatedbyeachteamaboutitsinnovation,pre-recordedvideosofeachteam’sconceptpresentation,andpre-recordedvideosofeachteam’sresponsestoastandardizedsetofquestions.Scoringwasdonewithoutpanelistsknowingwhichofthe16groupswerecontrolortreatmentgroups.HighSchoolStudentsTreatmentoutperformedcontrolonallsevenindividualitemsscored.Fourofthesedifferenceswerestatisticallysignificant:insightintochallenge,clarityandrelevanceoftheproblem,problemsolvingstrategy,andthepotentialimpactoftheirproposal.Whilethedifferencesbetweencontrolandtreatmentontheotherthreeitemsscoreddidnotreachstatisticalsignificance,thetreatmentgroupdidhavehigherratingsthanthecontrolgrouponeach.Similarly,inthetotalweightedteaminnovationoutcomescore,whichusedanaverageweightedtotalscoreacrossallitemsforthecontrolgroupcomparedtotheaverageweightedtotalscoreacrossallitemsforthetreatmentgroup,thetreatmentgrouphadhigherratingsbutthedifferencedidnotreachstatisticalsignificance.EarlyCareerSTEMProfessionalsNoneofthedifferencesbetweencontrolandtreatmentonthesevenindividualitemsscoredwerestatisticallysignificantfortheearlycareerSTEMprofessionals.Similarly,therewasnostatisticallysignificantdifferencebetweenthecontrolandtreatmentgroupsinthetotalweightedscoreacrossallitems.Hypothesis3SummaryItwasaveryimportantresultthattheexpertpanelistsratedthehighschoolproducts,processesandservicesofthetreatmentteamssignificantlyhigherthatthoseofthecontrolteamsintermsofinsight,clarity,problemsolvingstrategyandpotentialimpact.Itispossiblethatthislackoffindingsfromtheadultteamsmayresultfromusingacurriculumthatwasdevelopedspecificallyforadolescents.Furtherstudytodeterminewhetheradultfindingswouldchangewiththesubstitutionofacurriculumspecificallydesignedforusewithadultswouldbeveryuseful.RESEARCHCONCLUSIONS:Thestudylookedtoidentifydifferencesincreativityandcollaborationwhenusinganarts-basedapproachtograpplingwithlocalissuesandchallenges.Asnotedinthefindings,therewereanumberofpositivefindingsfromthestudy:
• Highschooltreatmentgroupsshowedalargenumberofstatisticallysignificantpositivedifferencesincreativethinkingskillsfrompretesttoposttest.Forthecontrolgroups,therewerenogainsonanyvariableafterthetraining.[Hypothesis1]
• Highschooltreatmentgroupsshowedstatisticallysignificantgainsinfouroffivedivergentthinkingskillsfrompretesttoposttest.Forthecontrolgroups,therewerenosuchgains.[Hypothesis1a]
AudienceViewpointsConsultingArtofScienceLearningResearchReport 10
• Highschooltreatmentgroupsshowedstatisticallysignificantgainsinthreeofsixconvergentthinkingskillsfrompretesttoposttest.Forthecontrolgroups,therewerenosuchgains.[Hypothesis1b]
• Highschooltreatmentgroupsshowedastatisticallysignificantpositivegainincriticalthinking
skillsfrompretesttoposttest.Thecontrolgroupsshowednosuchgain.[Hypothesis1c]
• Highschooltreatmentgroupsshowedsignificantlystrongerperformancethancontrolgroupsinsharingleadership,trustinmovingtowardasolution,transparencyincommunication,emotionallyintelligentbehavior,productivedisagreement,anddefiningacommonpurpose,basedonobservationaldata.[Hypothesis2]
• Highschoolstudentsperceivedtheirowncollaborativebehaviorshavingpositiveincreasesoverthetrainingforallofthemeasures.[Hypothesis2]
• AdultearlycareerSTEMprofessionalgroupsshowedsignificantlystrongerperformancethancontrolgroupsinemotionallyintelligentbehavior,mutualrespect,activefollowing,trustinmovingtowardasolutionandtransparencyincommunication,basedonobservationaldata.[Hypothesis2]
• EarlySTEMcareerprofessionalsperceivedtheirowncollaborativebehaviorsashavingpositiveincreasesoverthetrainingforalmostallofthemeasures.[Hypothesis2]
• Highschooltreatmentgroupsdevelopedsignificantlystrongerfinalinnovationoutputsthanthecontrolgroups.Externaljudgesfoundlargeandsignificantpositivedifferencesbetweencontrolandtreatmentgroupsininsightsintothechallenge,analyticclarity,problemsolvingstrategyandpotentialimpact.[Hypothesis3]
• Highschoolstudenttreatmentgroupsreportedasignificantlygreaterincidenceofapplyingtheirinnovationlearningexperiencestowork,school,volunteerandextracurricularactivitiesthanthehighschoolstudentcontrolgroups.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 11
DiscussionandImplicationsforFutureResearchTherewasatrendofastrongeroverallimpactofthetrainingonstudentsratherthanadults(especiallyinthecreativeandcollaborativethinkingself-reportmeasures),withthenotableexceptionofobservedcollaborativebehavior.Thereareseveralpossibleexplanationsforthistrend.Itmaybethatanarts-basedapproachtoinnovationtrainingismoreeffectiveforpeoplewhoarestillinschool,comparedtothosewhohavealreadystartedtheirSTEMcareers.Itisalsopossiblethatthedifferentagesandconsequentexperiencesoftheparticipantshadsomethingtodowiththedifferenceinresults.Athirdpossibleexplanationliesinthedecisionmadebytheresearcherstousethecurriculumdesignedforthehighschoolstudentswiththeadultparticipantsaswell,ratherthanmodifyingtheadultcurriculumtoreflectdifferencesbetweenthetwopopulations.Thisdecisionwasimportanttotheresearcherssincehavingverysimilar,ifnotidentical,approacheswasnecessaryinordertomakemoredirectcomparisonsbetweenthetwocohorts,aswellascombinethemasappropriatetotheanalyses.Ifthecurriculahadbeendifferentthenitwouldhavebeenimpossibletoknowwhetheranyvariationintheresultswereduetothedifferentaudiencesorthedifferentpedagogicalapproachesbeingused.Itwillbeimportanttotest,infuturestudies,whetherornotusinganarts-basedinnovationtrainingcurriculumspecificallydesignedforadultlearnerswouldleadtoastrongerimpactonthoselearners.ItisworthnotingthattherewereanumberofpositiveresultsfromthetreatmentapproachfortheearlycareerSTEMprofessionalswithrespecttotheobservedcollaborativebehaviors,whilethestrongerimpactwithstudentshappenedwithindividualcreativethinkingskillsandteaminnovationoutcomes.Itmaybethatthearts-basedapproachyieldedpositivedifferencesinhowtheadultsapproachedgroupinteractions,withouttranslatingintoachangeintheindividualsthemselvesorthesolutionstheycameupwith.Itwouldbeinterestingtoseeifanapproachtotheadultcurriculumthatallowsmorescopeforcollaborativeactivitythanwasincludedinthehighschoolcurriculum,mayproveeffectiveinyieldingpositivechangestoindividualcreativethinkingskillsorteaminnovationoutcomes.Itisalsopossiblethatthelackofdifferenceamongtheadultsinmuchofthevalidatedself-reportedmeasures(CPSP,CreativeProcesses)couldbearesultofthescalesbeingmoreusefulinothersituationsandthusnotasvalidforthistypeofapproach.Meanwhile,theobservationalmeasuresandrelatedweeklyself-reportedteamcollaborationmeasuresyieldedimpactsbetweencontrolandtreatmentconditions.Giventhattheseitemswerespecificallycreatedforthisstudy,itwouldbeexpectedthattheywouldyieldsomesignificantdifferences.Whiletheseinstrumentswerecarefullyplannedandcreated,theydidnotgothroughvalidityandreliabilitytesting,sofurthervalidationoftheinstrumentswouldbeusefultoseewhethertheseobservationalmeasureswouldbehelpfulinotherstudiesrelatedtocreativity,collaboration,andarts-basedapproachesinSTEM.Oneinterestingdifferenceisthatinmanycases,observations,activitytestsandteamoutcomesshoweddifferences,whilepreviouslyvalidatedself-perceivedmeasuresdidnotshowthesamedifferences.Asnotedinthereportandfindingsabove,thereweretimeswherethetreatmentorcontrolgroups(andinsomecases,boththetreatmentandcontrolgroups)showedadecreaseinoutcomemeasures,includingforbothcreativityandcollaboration.Theresearchersdonothaveasingleexplanationforthis;rather,therearesomedifferentpossibilities.Itcouldbethatthereissomethinginherentinthemeasuresthatimpactedthefindingsinthisparticularstudy,(i.e.,thestructuredlearningofinnovationprocessesthatformedthecoreofboththecontrolandtreatmentcurricula,ortheparticulararts-basedapproachusedinthetreatmentcurriculum).Therecouldalsobesomeunmeasureddifferencesunevenlydistributedbetweenthecontrolandtreatmentgroupsthatcontributedtothesedifferences,althoughatthispointthereisnotanyspecificevidencetowardswhattheymightbe.Variabilitywas
AudienceViewpointsConsultingArtofScienceLearningResearchReport 12
strongintermsofimpact,withhighstandarddeviationsatsomepoints.Thissuggeststherewasanunderlyingvariablenotmeasuredinpre-art,pre-scienceorcreativityscoresthatmighthelpdeterminewhetherthetrainingwouldhaveimpact.Futureresearchcoulddelveintothisarea.Additionally,theremaybesomethingaboutthetasksthemselvesthataccountforthedifferences.Withoutspecificevidenceoranindicationforwhichfactorsmayhaveinfluencedthisdecreaseinthetreatmentgroup,itisdifficulttospeculateaboutwhythismaybethecase.Furtherresearchcouldduplicatethestudytoseewhetherornotsimilarresultsforthetreatmentgrouparefound,allowingforabetterunderstandingofthisphenomenon.InlookingatpotentialrealworldapplicationsofthisresearchtothelearningandpracticeofSTEMinnovation,itseemsthatdifferentapproachestotheintegrationofarts-basedlearningmayworkbestfordifferentaudiences.Forhighschoolstudents,thearts-basedapproachwasclearlymoreeffectiveinfosteringcreativethinkingandproblemsolving,suggestingthatitmaybemoreeffectivethanatraditionalapproachforstudents.ThishaspotentiallyfarreachingimplicationsforhowadolescentsmightbestexperienceSTEMandinnovationlearninginawiderangeofformalandinformalsettings.Furtherresearchcouldinvestigatewhichspecificfactorswouldbemorelikelytoleadtopositiveoutcomes,andpotentialstrategiesforoptimizingtheimpactsofarts-basedlearning.Foradults,outcomessuggestthatanarts-basedapproachmaybemosteffectivewhenusedtoincreasecollaborationorstrengthenteamsandcollaborativecultureinaprofessionalsetting.Itwouldalsobevaluabletoexplorethepossibilitythatbuildingincreasedopportunitiesforcollaborativeexplorationintoanadultarts-basedcurriculummayyieldgreaterimpactonindividualcreativethinkingskillsandteaminnovationoutcomes.Itispossiblethattheconditionsthatneedtobepresenttopositivelyimpactadultcreativethinkingskillsandinnovationoutcomesmaybedifferentfromthoseneededforstudents.Theremayalsohavebeensomedifferentkindsofcreativethinkingskillsthatcouldbeenhancedwithanarts-basedapproach.Theseareimportantquestionsforfurtherinvestigation.InboththeadolescentandearlycareerSTEMprofessionalcases,largerstudieswithmorethanonetreatmentgroupforeachaudiencewouldbeinvaluable;thisseemsanobviousnextstepinbuildingonthisresearch.Giventhedearthofresearchinthisarea,andtheuniquenessofthearts-basedapproachtothelearningandpracticeofSTEMinnovation,itisnotsurprisingthatevenwiththecompellingandpromisingfindingsinthisstudythereareadecentnumberofunansweredquestions.Whileaquasi-experimentalapproachwasabletobeusedforthecurrentstudy,anexperimentalstudywouldgoevenfartherinansweringsomeofthemorefundamentalquestionsstillremainingaroundcreativityandcollaboration,manyofwhichhavebeentouchedonabove. Basedontheoutcomesofthisstudy,westronglyrecommendfutureresearchintothefollowingquestionsasparticularlyusefultotheadvancementofknowledgeinthisfield:• Doesarts-basedlearningfosterengagementinadolescentandadultlearners?Forbothcohorts,
theattritionratewasconsiderablyhigherforthecontrolgroupthanforthetreatmentgroup.Whiledatawerenotcollectedduringthisstudyaboutwhyparticipantswerenolongercontinuingwiththeprogram,onepossibleexplanationisthatthearts-basedapproachengenderedagreaterlevelofengagementthanthetraditionalapproach.Thiswouldbeinterestingtopursueinfurtherstudies.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 13
• Howcanarts-basedlearningbemosteffectivelyintegratedintoinnovationtraining?Weobservedmanypowerfulpositiveimpactsofthistypeofinnovationtraining,whencomparedwithamoretraditionalpedagogicalapproach.o Arethereparticulararts-basedactivitiesandapplicationsthatdrovetheimpactsoncreative
thinkingskills,collaborativebehaviorsandinnovationoutcomesthatweobservedandmeasured?Mighttheuseofothersuchactivitiesfurtherstrengthenorbroadenthoseimpacts?
o Whichtypesandwhatamountsofarts-basedlearning,andinwhatsequence,wouldoptimizeimpact?
• Canarts-basedlearningservetoneutralizethepotentialnegativeimpactsoftraditionalhigh
schoolinnovationtrainingoncreativity?Thehighschoolstudenttreatmentgrouphad7creativethinkingskillvariablesthatshowedasignificantincreasefrompretopost,andnoneoftheseweresignificantlydifferentforthecontrolgroup.Likewise,thehighschoolstudentcontrolgrouphad5creativethinkingskillvariablesthatshowedasignificantdecreasefrompretopost,yettherewasnosignificantdecreaseforthetreatmentgroup.Theseresultsstronglysuggestthatarts-basedlearningmayhavethecapacitytoovercomeandneutralizewhatappearstobethenegativeimpactoftraditionalinnovationtrainingonadolescentcreativethinking.Boththequestionofwhethertraditionalinnovationtrainingdepressescreativityinadultlearners,andthepotentialofarts-basedlearningtoovercomeanynegativeimpact,bearsfurtherinvestigation.
• Doestheuseofarts-basedlearningenhancethecognitivegainsofadolescentoradultlearners?Thisstudydidnotexaminetheextenttowhichthearts-basedapproachtoinnovationtrainingincreasesknowledgeandunderstandingofSTEMcontent,bothwithrespecttolearningingeneralandforspecifictopics.Severalofourfindings,however,suggestthatthismaybeapossibility.Thisisanimportanttopictopursueinfuturestudies.
• Doesarts-basedlearningfostergreaterpost-learningimpact,agencyandoptimisminadolescentlearners?Inasurveyfourmonthsaftertheintervention,thehighschoolstudenttreatmentgroupreportedasignificantlygreaterincidenceofapplyingtheirinnovationlearningexperiencestowork,school,volunteerandextracurricularactivitiesthanthehighschoolstudentcontrolgroup.Thetreatmentgroupalsoshowedamuchgreateroptimismthattheywouldbelikelytoapplytheirinnovationlearningexperiencestofuturework,school,volunteerandextracurricularactivitiesthanthehighschoolcontrolgroup.Thesefindingssuggestthevalueofinvestigatingtheimpactofarts-basedlearningonadolescentagencyandoptimism.
• Isself-reportanaccuratewaytoassesscollaborationingroups?Althoughneithercontrolnortreatmentgroupsratedthemselvesashavingimprovedonanyoftheself-reportedteamratingmeasures,observersrecordedthetreatmentgroupsashavingengagedinmanyofthesebehaviorssignificantlymorebythelastsession.Thesefindingssuggesttheimportanceofincludingdirectobservationinassessmentsofthenatureandextentofgroupcollaboration,ratherthansimplyrelyingonself-reportfromparticipants.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 14
IntroductionandProjectBackgroundTheArtofScienceLearningPhase2isafour-yearNationalScienceFoundation-fundedinitiative(Grant#1224111)thatusestheartstosparkcreativityinscienceeducationandthedevelopmentofaninnovative21stCenturySTEMworkforce.TheinitiativeisbuiltonArtofScienceLearning’sPhase1,aswellasdecadesofworkbyProjectDirectorHarveySeifterandcolleagues,exploringtheimpactofartisticskills,processesandexperiencesoncreativity,innovationandlearning(aneducationalmethodologyknownas“arts-basedlearning”).Thegoaloftheproject'sPhase2developmentactivitieswastoexperimentwitharts-basedlearninginavarietyof"innovationincubator"modelsincitiesaroundthecountry.Modeledonbusiness"incubators"or"accelerators"thataredesignedtofosterandaccelerateinnovationandcreativity,theseSTEMincubatorsgeneratecollaborationsofdifferentprofessionalsandthepublicaroundSTEMeducationandotherSTEM-relatedtopicsoflocalinterestthatcanbeexploredwiththehelpofcreativelearningmethodologies.Theseincludeinnovativemethodstogeneratecreativeideas,ideasfortransformingoneSTEMideatoothers,drawingonvisualandgraphicalideas,improvisation,narrativewriting,andtheprocessofusinginnovativevisualdisplaysofinformationforcreatingvisualroadmaps.TheArtofScienceLearning,incollaborationwithBalboaParkCulturalPartnership,Phase2’snationaladministrativesponsor,andseveralinformalscienceeducationandotherculturalandbusinessorganizationsinSanDiego,Chicago(notablytheMuseumofScienceandIndustry),andWorcester,MA(principallyEcoTariumandClarkUniversity)implementedaresearchanddevelopmentprojecttoinvestigatearangeofpossibleapproachesforstimulatingthedevelopmentof21stCenturycreativityskillsandinnovativeprocessesattheinterfacebetweeninformalSTEMlearningandmethodsforcreativethinking.Theprojectgoalsincludeddevelopingnewwaystousearts-basedlearningtoenhanceSTEMinnovationskillsamongarangeoflearners,developingimpactfulnewmodelsusingarts-basedlearningtostrengtheninformalSTEMlearningandadvanceunderstandingofthepotentialimpactsofarts-basedlearningonthepublic'sunderstandingofandengagementwithSTEM.Aparticularfocuswasonstrengtheningthe21stCenturyworkforceskillsofhighschoolstudentsandearlycareerSTEMprofessionals,andcreatingdeliverablesandresourcesforthefieldtofostertheintegrationoftheartsandSTEMfields.Activitieshaveincludedthefollowing:theformationandcollaborativeprocessesoftwoincubatorsites,anindependentresearchstudy(thisreport),thedevelopmentofacomprehensivearts-basedinnovationprocesscurriculumforSTEMlearners,professionaldevelopmentbasedonthecurriculum,publicengagementeventsandexhibits,aprojectwebsiteandtoolsforsocialnetworking,andprojectevaluation.Anationaladvisorycouncilincludedprofessionalsineducation,science,creativity,andbusiness.Formoreinformationabouttheprojectseethefollowing:
• ArtofSciencelearningwebsite:http://www.artofsciencelearning.org/• NSF’swebsiteabouttheprojectat
http://www.nsf.gov/awardsearch/showAward?AWD_ID=1224111&HistoricalAwards=false• Theprojectsiteontheinformalscience.orgsiteathttp://informalscience.org/projects/ic-000-
000-000-099/Integrating_Informal_STEM_and_Arts-Based_Learning_to_Foster_Innovation
Asmentionedabove,theprojectincludedasummativeevaluationoftheincubatorsbySlover-Linnett(aprofessionalevaluationfirm)thatfocusedonthegoalsandobjectivesoftheproject.ThecurrentreportsummarizestheseparateresearcheffortconductedbyAudienceViewpointsConsulting(AVC)that
AudienceViewpointsConsultingArtofScienceLearningResearchReport 15
lookedintobetterunderstandingthetheorybehindtheintersectionofartsandsciencesincollaborativeprocessesfocusedonsolvingreal-worldproblems.AVCwascontractedtoconductindependentresearchstudiesatsitesinWorcester,Massachusetts(attheEcoTarium)andSanDiego,California(attheBalboaParkCulturalPartnership).ThemaingoaloftheresearchwastobetterunderstandtheaffordancesofusingacurriculumthatintegratedtheartsintoSTEMinnovation,includingtheuseofarts-basedprocessesandpracticesintryingtosolvespecificproblemsor‘challenges’identifiedinthecommunity.TheresearchfocusedonbothhighschoolstudentsandearlycareerSTEMprofessionalsinordertobetterunderstandtheconditionsandfactorsrelatedtospecificoutcomes.HighschoolstudentswereincludedintheresearchsincetheyareapopulationthatislearningaboutandbecomingmoreinterestedinSTEM,andtheseareformativeyearsforthinkingaboutandengaginginthesetopics.TheearlycareerSTEMprofessionalswereincludedinordertogatherinformationaboutthepotentialforanarts-basedapproachtobeusedwiththosejustbeginningorcontinuingtoengageinSTEM-basedfields.Includingbothwouldallowforabetterunderstandingofthepotentialofengaginganarts-basedapproachtocollaborativeproblemsolvingindifferentaudiences.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 16
StudyDesignAoSL’sresearchcomponentwasdesignedtotesttheideathatintegratingtheartsintoSTEM-relatedinnovationtrainingwouldresultinenhancedcreativethinkingskills,moreextensivecollaboration,morerobustinnovationprocessesandimprovedinnovationoutcomes.Twocities,Worcester,MassachusettsandSanDiego,California,servedasthesitesfortheresearchstudy.HighschoolstudentswerethesamplepopulationinMassachusetts,andearlycareerSTEMprofessionalsinCalifornia.AtbothsitestheAoSLprojectteamhostedfiveweek-longinnovationtrainingsessions,witheachgroupmeetingforahalfdayperweek,totalingroughly20hoursofinvolvementintheproject.Thetrainingsessionsinvolvedproject-basedlearningfocusedonthefrontendofinnovation,withprojectsaddressinglocalSTEMchallenges(transportationalternativesinWorcester,waterresourcesinSanDiego).Overthecourseofthefiveweeks,teamsofparticipantscreatedsimpleprototypesandbusinesscasesfornewproducts,processesandservicestoaddressthesechallenges.Thetrainingcurriculum,groundedinbestpracticesderivedfromtheProductDevelopmentManagementAssociationBodyofKnowledge,includedkeyconceptsofinnovation,STEMcontentspecifictoeachlocalchallengeandcollaborativeprojectinnovationactivitiesandexercises.Thedecisionwasmadetousethecurriculumdesignedforusewithadolescentsforbothgroups,inordertoallowforacomparisonbetweenthehighschoolstudentsandearlycareerSTEMprofessionals,aswellascombinethemasappropriatetotheanalyses.Withoutusingthesamecurriculum,itwouldhavebeenimpossibletodeterminewhetheranypositiveresultsweregeneralizableacrossdifferentaudiences.Belowisthebasicscheduleforeachworkshopday,comparingactivitiesanddiscussionbetweenthecontrolgroupandthetreatmentgroup(seeTable1).TheworkshopstaffutilizedthesamecurriculumandmaintainedthesameschedulefortheWorcestercohortandtheSanDiegocohortinordertocomparedifferencesbetweenthegroupsofadultsandstudents.Thestudyusedaquasi-experimentaldesignwithpre-test,post-testintactgroupdesign,inordertoallowcontrolforothervariables(seeFigure1belowforthedesignandmethodsforthecontrolandtreatmentgroups).Thecontrolgroupswereincludedtoprovideacomparisontotreatmentgroupparticipants,whoexperiencedthearts-basedactivities.BothgroupsexperiencedaSTEMinnovationworkshop;however,thetreatmentgroupalsoexperiencedthearts-basedactivities,whichallowedforisolatingandunderstandingtheaddedvalueofthearts-basedactivitiestotheworkshopexperience.Thestudyalsoincorporatedmixedmethods;whilethefocuswasonquantitativedata,criticalqualitativedatawerecapturedtobetterunderstandtheindividualexperienceofparticipatingintheInnovationChallenges.Therewerethreemainhypotheses,whichincludedsub-hypotheses:
1. Artsbasedlearninginfluencesindividuals’creativethinkingskills.a. Arts-basedinnovationtrainingincreasesanindividual’sabilitytoemployconvergent
thinkingcomparedtotraditionalinnovationtraining.b. Arts-basedinnovationtrainingincreasesanindividual’sabilitytoemploydivergent
thinkingcomparedtotraditionalinnovationtraining.c. Individualswhoparticipateinarts-basedinnovationtraininghaveagreaterincreasein
criticalthinkingskillsthanthosewhoparticipateintraditionalinnovationtraining.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 17
2. Artsbasedlearninginfluencesinnovation-relatedprocessbehaviorswithintheteamcontext.a. Artsbasedinnovationtrainingincreasesanindividual’scollaborativebehaviorsand
collaborativeskills,ascomparedwithtraditionalinnovationtraining.
3. Artsbasedlearninginfluencesthequalityofthefinalproductdevelopedbytheteam.a. Artsbasedinnovationtraining,comparedtotraditionalinnovationtraining,willresultin
ateamproducingamoreinnovativeandfeasibleproduct.
BothhighschoolstudentsandtheearlycareerSTEMprofessionalsmaintainedthesameschedule,whilethereweredifferencesintheactivityflowsofthecurriculumbetweenthecontrolandtreatmentgrouptoaccountforreplacementof9hoursofthetraditionalinnovationpedagogyusedinthecontrolcurriculumwith9hoursofarts-basedactivitiesdesignedtoachievethesamelearningobjectivesusedinthetreatment(seeTable1).Table1:ResearchCurriculumSchedule
Morning-TreatmentGroup Afternoon-ControlGroup
R1 1.Welcomeandoverview9:00am-9:15am
1.Welcomeandoverview1:30pm-1:45pm
2.CreativitySkillsTest9:15am-9:30am 2.CreativitySkillsTest1:45pm-2:00pm3.IntroductiontoChallenge9:30am-10:30am
3.IntroductiontoChallenge2:00pm-3:00pm
4.IntroductiontoInnovation10:30am-10:50am
4.IntroductiontoInnovation3:00pm-4:15pm
5.Metaphormingactivity11:05am-12:50pm
5.InnovationTools&TheirUse4:15pm-5:15pm
6.Wrapup12:50pm-1:00pm 6.Wrapup5:15pm-5:30pmR2 1.Welcomeandreview9:00am-9:15am 1.Welcomeandreview1:30pm-1:45pm
2.IntroductiontoOpportunity9:15am-9:30am
2.IntroductiontoOpportunity1:45pm-2:00pm
3.InformationGathering/Enrichment#19:30am-10:15am
3.ExpertPanel2:00pm-2:30pm
4.ExpertPanel10:15am-10:45am
4.Q&AwithExpertPanel2:30pm-3:00pm
5.Q&AwithExpertPanel11:00am-11:30am
5.BrainstormingThreeOpportunitySources3:00pm-3:30pm
6.Problem/OpportunityWorkshop11:30am-11:45am
6.IdentifyingOpportunitiesandBuildingSharedGoals3:45pm-4:40pm
7.Problem/OpportunitySelectionWorkshop11:45am-12:50pm
7.DescribeWhatProblems/SolutionsYourTeamSelected4:40pm-5:10pm
8.Wrapup12:50pm-1:00pm 8.Wrapup5:10pm-5:30pm
AudienceViewpointsConsultingArtofScienceLearningResearchReport 18
R31.Welcomeandreview9:00am-9:05am 1.Welcomeandreview1:30pm-1:45pm
2.OpportunitySelection9:05am-9:50am 2.Vision2025Exercise1:45pm-2:45pm
3.IdeationCycle9:50am-10:55am 3.SolutionSelection2:45pm-3:30pm4.IdeationCycle,parttwo
10:55am-11:40am4.ResearchingProblem/Solution
Statements3:45pm-4:45pm5.IdeaEnrichmentandRefinement
11:40am-12:55pm5.ScoringEnhancedSolutions
4:45pm-5:10pm
6.Wrapup12:55pm-1:00pm 6.Wrapup5:10pm-5:30pmR4
1.WelcomeandReview9:00am-9:05am 1.Welcomeandreview1:30pm-1:45pm2.ConceptSelection9:05am-9:40am
2.PrioritizeProduct/ServiceConceptsfromsolutions1:45pm-3:15pm
3.RehearsingIdeaSession9:40am-11:30am
3.ConceptFeasibilityandMarketEnhancement3:30pm-4:30pm
4.CreatingBusinessCases11:30am-12:55pm
4.BeginBusinessCaseDevelopment4:30pm-5:15pm
5.Wrapup12:55pm-1:00pm
5.Wrapup5:15pm-5:30pm
R51.WelcomeandReview9:00am-9:05am 1.Welcomeandreview1:30pm-1:35pm2.CompletingBusinessCases
9:05am-9:45am2.Iteration&BusinessCaseReview
Discussion1:35pm-1:50pm
3.PresentationDevelopment9:45am-10:40am
3.WriteClearandComprehensiveConcept
Statements1:50pm-2:10pm4.CastingandRehearsingthe
Presentations10:40am-11:45am4.Sub-groupsWorkonBusinessCasesandGavePresentations2:10pm-4:15pm
5.GatePresentations11:45am-12:45pm 5.GatePresentations4:15pm-5:15pm6.CreativitySkillsTest12:45pm-1:00pm
6.CreativitySkillsTest5:15pm-5:30pm
Note:Therewasa15minutebreakroughlyinthemiddleofeachfour-hoursession,forbothcontrolandtreatmentgroups.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 19
Figure1:StudyDesignandMethodsVisualRepresentation
HighSchoolStudents
Control Treatment
RecruitmentPhase• PreSurveyInterventionWeek1• CreativitySkillsPreTest
• ResearcherObservation
• CollaborationRatings
Weeks2-4• ResearcherObservation
• CollaborationRatings
Week5• ResearcherObservation
• CollaborationRatings
• FinalPresentation
• CreativitySkillsPostTest
PostIntervention• PostSurvey• TransferabilityStudy
EarlySTEMCareer
Professionals
Control Treatment
AudienceViewpointsConsultingArtofScienceLearningResearchReport 20
MethodsThestudyusedanexperimentaldesignwithtreatmentandcontrolgroups;controlgroupswereincludedtoprovideacomparisontotreatmentgroupparticipantswhoexperiencedthearts-basedactivities.BothgroupsexperiencedaSTEMinnovationworkshop;however,thetreatmentgroupalsoexperiencedthecreativethinkingactivities,allowingforanunderstandingoftheaddedvalueofthearts-basedactivitiestotheworkshopexperience.Thestudyalsoincorporatedmixedmethods;whilethefocuswasonquantitativedata,criticalqualitativedatawerealsocapturedtobetterunderstandtheindividualexperienceofparticipatingintheInnovationChallenge(seeTable2).Datawerecollectedattwolocations:theEcoTariuminWorcester,Massachusetts,andBalboaParkinSanDiego,California.Furthermore,thereweretwomainaudiencesthestudyfocusedon:1)highschoolstudents,and2)adultswhowereatanearlystageintheirSTEMcareers(earlycareerSTEMprofessionals).AllparticipantsintheWorcestercohortwerehighschoolstudents,andalloftheparticipantsintheSanDiegocohortwereearlycareerSTEMadults.Atotalof65individualsparticipatedinthecompletecourseof5Worcesterworkshops,andatotalof69individualsparticipatedinthecompletecourseof5SanDiegoworkshops.WhileparticipantswereawarethattheywereparticipatinginanNSF-fundedstudy,theywerenotawarethatitwasanexperimentalstudyorthatitwaslookingatarts-basedapproachestoinnovation.InlookingatattendanceratesfortheWorcestercohort,therewasaninterestingeffect-thetreatmentgroupsaw32ofthe34initialparticipantscompletethefivesessions,anattritionrateof6%;whilethecontrolgroupsaw33oftheinitial38participantscompletethefivesessions,anattritionrateof13%.TheSanDiegogroupssawasimilarpatternofattrition:37ofthe40initialparticipantsinthetreatmentgroupcompletedallfivesessions,anattritionrateof8%;whilethecontrolgroupsaw32oftheinitial36participantscompletethefivesessions,anattritionrateof11%.Forbothcohorts,theattritionratewasconsiderablyhigherforthecontrolgroupthanforthetreatmentgroup,thoughthisoccurredtoagreaterdegreeinWorcester.Ineachcohort,thegreaterlevelofattritionamongthecontrolgroupswasmanifestbyweektwo.Whiledatawerenotcollectedduringthisstudyaboutwhyparticipantswerenolongercontinuingwiththeprogram,onepossibleexplanationisthatthecontrolandtreatmentapproacheswereengenderingadifferentlevelofengagement.Multiplemethodswereusedaspartofthislargeresearchplan,including:
1. RecruitmentSurvey2. Pre-andpost-interventionwrittenquestionnaires;3. Creativethinkingskillsassessment;4. Observations;5. Follow-uptransferabilitysurveyswithasubsetofparticipants;and6. Scoringofinnovationteamproducts.
SeetheAppendicesofthereportfortherelatedinstruments.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 21
Table2:SummaryofMethods
Method WhenAdministered SourceofApproach,InstrumentorItems
RecruitmentSurvey(DemographicVariables)
BeforeParticipation(Pre)
Researcher-generatedbasedonprojectneeds.
PreWorkshopSurvey(CurrentlyLevelofArtsandSTEMparticipation)
BeforeParticipation(Pre)
ModifiedfromSlover-Linnett’sevaluationoftheAoSLincubators.
CreativeThinkingSkillsTestSurvey
BeforeandAfterParticipation(Pre/Post)
AnactivitydevelopedbyHarveySeifter,modifiedfromRuncoandBasadur’s(1993)articleAssessingIdeationalandEvaluativeSkillsandCreativeStylesandAttitudes,andscoredonarubricdevelopedbyHarveySeifterandAVC.
Slover-LinnettCreativeProcessPerceptionsSurvey
BeforeandAfterParticipation(Pre/Post)
ScaledevelopedbySlover-Linnett(AoSLevaluator)tolookatperceptionsofthecreativeprocess.Itworkedwellontheevaluationpretest,providingasignificantrangeofopinions.
Basadur’sCreativeProblemSolvingProfile(CPSP)Survey
BeforeandAfterParticipation(Pre/Post)
AmodifiedversionofBasadur,Graen,andWakabayashi’s(1990)CreativeProblemSolvingProfile(CPSP)Inventory,whichmeasurespreferencefordifferentstagesinthecreativeprocess,differentiatinghowindividualsprefertogainanduseknowledge.Theyproposefourstagestoanindividual’screativeprocess.
CriticalThinkingSurvey
BeforeandAfterParticipation(Pre/Post)
DrawnfromBasadurandFinkbeiner’s(1985)scaleonpreferenceforideationandtendencyforprematurecriticalevaluationofideas,thisscalewasusedinRuncoandBasadur’s(1993)studyonideationandevaluativethinkingchangeduringtraining.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 22
CreativeCompetenciesSurvey
BeforeandAfterParticipation(Pre/Post)
AdaptedfromanabridgedversionoftheEpsteinCreativityCompetenciesInventoryforIndividuals(ECCI-i),developedbyEpstein,Schmidt,andWarfel(2008).Therearefourmeasurable,trainablecompetencies:capturing(preservingnewideasastheyoccur),challenging(takingondifficulttasks),broadening(seekingknowledgeandskillsoutsideone'scurrentareasofexpertise),andsurrounding(seekingoutnewstimuliorcombinationsofstimuli).
Self-ratedCommonPurposeandCollaborationSurvey
DuringParticipation(EachWeek)
Eleventraitsofcollaborativebehavior,identifiedbytheprojectdirector,weredrawnfromthecollectiveworkofHackman(2002),KatzenbachandSmith(1993),Sawyer(2007),SengeandScharmer(2001),ThotaandMunir(2012),MossKanter,HeifetzandLinsky(2002).Individualswereaskedtoratetheirteameachweek;thesametraitswereincludedfortheObservedBehaviorsmethod.
ObservedBehaviors DuringParticipation(EachWeek)
Eleventraitsofcollaborativebehavior,identifiedbytheprojectdirector,drawingfromthecollectiveworkofHackman(2002),KatzenbachandSmith(1993),Sawyer(2007),SengeandScharmer(2001),ThotaandMunir(2012),MossKanter,HeifetzandLinsky(2002).Theresearcherscountedinstancesofthevariousbehaviorsduringcollaborativetaskseachweek;thesametraitswereincludedfortheSelf-ratedCommonPurposeandCollaborationSurvey.
TeamInnovationOutcomesAssessmentPanel
AfterParticipation(Post)
ExpertpanelistsfromtheProductDevelopmentManagementAssociation’sselectioncommitteeofOutstandingCorporateInnovationAwardsviewedteams’finalproducts(businesscase,PowerPointpresentations,videoofconceptpresentation,andvideoofresponsestoquestions.Thepaneldevelopedarubricwithsevenelements,ratedeachteamindependentlyontheseelements,thencalculatedafinalscoreforeachteam.
RecruitmentSurveyAstheteamwasattemptingtorecruitindividualswhometadefinedsetofcriteria,potentialrecruitswereaskedtofilloutaquestionnairetofindoutmoreabouttheirbackgroundandrelevantexperiences.ThequestionnaireaskedaboutinvolvementwithSTEMactivities,currentemploymentstatus,currentstudentstatus,genderandage.ThequestionnairealsoaskedhowparticipantsfoundoutaboutthisInnovationChallengeresearchproject.Thequestionnairecapturedthecontactinformationfor
AudienceViewpointsConsultingArtofScienceLearningResearchReport 23
participantsinordertoemailinformationandaninvitationtothepreworkshopsurvey.ThisquestionnairewaspostedonapublicwebpageusingtheSurveyMonkeyweb-basedplatformtoprovideaccesstothepotentialrecruits.
PreWorkshopSurveyOnceacceptedintotheInnovationChallenge(i.e.,theworkshop),participantswereaskedtocompleteasurveypriortotheChallengestart,inordertolearnmoreabouttheparticipantandsothatresponsestothispre-surveycouldbecomparedtotheresponsestothepost-surveyaftercompletingtheInnovationChallenge.Thequestionswerebrokenupintosectionsthatdirectlycorrelatedtothemainhypothesestestedwithinthestudy.AlinktothesurveywassentviaemailtoparticipantsandthesurveywashostedonaprivatewebpageontheSurveyMonkeyweb-basedplatform.SeeAppendixA.Question1collectedinformationaboutanindividual’spriorconnectionswithartandscience.Thisinformationwasmostlyusedtosortindividualsintoteamssothattheteamshadsimilaraveragescoresforartandscienceexperience,toreducebiasbetweenteams.Question2focusedoncreativethinking.ThequestionwasadaptedfromanabridgedversionoftheEpsteinCreativityCompetenciesInventoryforIndividuals(ECCI-i),developedbyEpstein,Schmidt,andWarfel(2008).Slover-Linettusedthisscalewithsomesuccesswithinthepre-surveyforincubatorparticipants.Question3wasadaptedfromtheNorton(1975)ToleranceforAmbiguityScale.TheAVCresearchteamtookitemsfromthe“problem-solving”and“philosophy”subscales;thereweresixothersubscales.Toleranceforambiguitymaybeasignificantindependentvariableforparticipants,giventhenatureofthecontrolandtreatmentcurriculum.Question4wasdevelopedbySlover-LinnetttolookatperceptionsoftheCreativeProcess.ItseemedtofunctionwellontheirpretestwithasignificantrangeofopinionsQuestion5wasdesignedtolearnmoreaboutcriticalthinkingskills.DrawnfromBasadurandFinkbeiner’s1985scaleonpreferenceforideationandtendencyforprematurecriticalevaluationofideas,thisscalewasusedinRuncoandBasadur’s(1993)studyonideationandevaluativethinkingchangeduringtraining.Question6wasdrawnfromamodifiedversionofBasadur,Graen,andWakabayashi’s(1990)CreativeProblemSolvingProfile(CPSP)Inventory,whichmeasurespreferencefordifferentstagesinthecreativeprocess,differentiatinghowindividualsprefertogainanduseknowledge.Participantswerethenaskediftheyidentifiedasaninnovator,andpromptedtogivetheirownpersonaldefinitionofinnovation.Thesurveyconcludedwithafewadditionaldemographicquestions,likeethnicoriginandhighestlevelofeducationcompleted.
PreWorkshopCreativeThinkingSkillsTestParticipantsintheInnovationChallengewereaskedtocompletean“InnovationWarm-up”exercise,atthestartofthefirstdayoftheworkshop.ThetaskincludeddefiningproblemsandchoosingaproblemrelatedtotheInnovationChallengeasdescribedbythepromotionalmaterial,thendefiningsolutions
AudienceViewpointsConsultingArtofScienceLearningResearchReport 24
andchoosingasolutiontothepreviouslychosenproblem.Participantshad15minutestocompletetheskillstest.Thistestwastakenwithpaperandpencil.SeeAppendixFandAppendixG.ThesetaskswerederivedfromconceptssimilartothechallengesnotedinRuncoandBasadur’s(1993)articleonAssessingIdeationalandEvaluativeSkillsandCreativeStylesandAttitudes.Thesolutionswerescoredinacomplexrubricincludingthequantityofproblemsandsolutionsidentified,thebreadthofproblemsandsolutions,andthemotivationsforchoosingaparticularproblemtofocuson.TherubriccanbeseeninAppendixI.
WorkshopObservationsDatacollectorsobservedparticipantseachweekoftheInnovationChallenge.Thepurposeofthisapproachwastodocumentexamplesofindividualcollaborativebehaviorsinteamsettings,acrossthefiveresearchsessionsorweeks.AVCscheduledatleasttwodatacollectorsforeachworkshopweek;AVCteammemberscollectedsomedata,butalsohiredlocalindividualswhohadexperienceobservingbehaviorinanexperimentsetting.Duetotheconstraintsofthenumberofgroupsineachworkshopandthenumberofdatacollectorsitwasimpossibletoobserveeachgroupduringallactivitiesandgroupdiscussionperiods.Itwasthereforenecessaryfordatacollectorstorotatebetweengroupsduringtheworkshopsessions.SeeAppendixC.Datacollectorsobservedtheindividualsononeteamforaperiodoftime,usuallybetween10and30minutes,duringagroupactivityordiscussionperiod.Sincethebehaviorsfocusedoncollaboration,participantswerenotobservedduringpresentationsbyfacultyorwhenindividualswereworkingontheirown.TheAVCandArtofScienceLearningteamsdevelopedalistofbehaviorstobeobservedwhichcorrespondedtothemainhypothesestestedwithinthestudy.Datacollectorsrecordedbehaviorsdisplayedbyindividualsontheteamduringtheobservationperiod,andadditionallyrecordedanyunusualornoteworthyactivityoutsideofthelistofbehaviorstoobserve.Observationswererecordedonpaperdatasheetswithpenorpencil.Attheendofthepredeterminedobservationperiodthedatacollectorsmovedontothenextteamandbeganwithanewobservationperiod.Eachsessionwasdiscussedatlengthbeforethesessionstodeterminewhentheparticipantswouldbeobserved,forhowlong,andwhendatacollectorswouldswitchbetweenteams.Withtwodatacollectorsandfourteams,switchingoncewithinanobservationperiodwouldmeanthatallfourteamswereobservedduringanactivityorobservationperiod.
TeamCollaborationRatingsAttheendofeachworkshopsessionparticipantswereaskedtoratetheirteamonaseriesofbehaviorsthatdirectlyalignedwiththecollaborativebehaviorsrecordedbydatacollectorsintheworkshopobservationsheet.Thiswasanattempttotriangulatearealisticdepictionofanindividual’scollaborationandparticipationintheInnovationChallenge.Individualswereaskedtoratetheirteam’sperformancebasedonaseriesofbehaviorsonascalefrom1to7points,andthenwereaskedtogivetheirteamanoverallratingbetween1and7points.ThesurveysweretakenoniPadsthroughtheSurveyMonkeyweb-basedplatform.Therewere4to8iPadsavailableduringtheworkshops,soindividualstookturnsfillingoutthesurveybeforesubmittingtheirsurveyandpassingtheiPadontothenextteammember.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 25
PostWorkshopCreativeThinkingSkillsTestJustbeforethecloseofthelastdayoftheInnovationChallenge,participantswereaskedtoagaincompletetheexercisetheyhaddoneontheirfirstdayoftheworkshopthistimedescribedasan“InnovationWrapUp.”Theformatoftheexercisewasexactlythesameasthefirstsession:theparticipantsinbothcohortswereaskedtoaddressaccesstoaffordableandhealthyfood,ratherthanthetopicsthesecohortshadbeenworkingonoverthepreviousweeks.SeeAppendixH.
PostWorkshopSurveyApproximatelyoneweekaftercompletingtheInnovationChallenge,participantsweresentalinktoaPostWorkshopSurvey,whichwashostedonaprivatewebpagethroughtheSurveyMonkeyweb-basedplatform.ThissurveyhadtheexactsamequestionsasthePreWorkshopSurvey,Questions2to6,inordertocomparetheresponsesofindividualsbeforeandaftertheworkshopexperience,andtocomparetheresponsesbetweenthecontrolandtreatmentgroups.SeeAppendixB.Again,likethePreWorkshopSurvey,individualswereaskediftheyself-identifiedasaninnovator;however,thistimeindividualswerealsoaskediftheirdefinitionofinnovationhadchangedsinceparticipatingintheworkshop.Finally,participantswerealsoaskedtorateeachindividualmemberoftheteam,onascalefrom1to7,basedontheircontributionsoverthefivesessions.
TransferabilitySurveyAboutfourmonthsaftercompletingtheInnovationChallengeAVCsentafollowupwebsurveytoallparticipantstoexplorethelong-termeffectsoftheirexperience.ThesurveywashostedonaprivatewebpagethroughtheSurveyMonkeyweb-basedplatform.Participantswereaskedwhethertheyhadbeenthinkingabouttheworkshopsinceitscompletion,iftheworkshophadchangedhowtheythinkaboutthecreativethinkingprocess,andwhetherindividualsthoughttheywouldbeabletousesomeofthetechniqueslearnedduringtheworkshopsintheirprofessionalandpersonallife.CompletionoftheTransferabilitysurveywasnotmandatory;therefore,thetotalnumberofindividualswhoparticipatedinthismethodislowerthanthetotalnumberofindividualswhoparticipatedintheworkshops.SeeAppendixE.
TeamInnovationOutcomesAssessmentPanelRatingsThereviewofthefinalteaminnovationoutcomeswasconductedbyapanelcomposedofthreemembersoftheselectioncommitteeoftheProductDevelopmentManagementAssociation’s(PDMA)OutstandingCorporateInnovationAwards,includingthatcommittee’sfoundingandcurrentChairs.Thepanel,consultingwiththeprojectandresearchteams,developedanassessmentrubricidentifyingandweightingsevenmeasurestogaugethequalityoftheinnovationoutcomes,includingteaminnovationoutputsandworkproducts.Theysubsequentlyappliedtherubrictoeachteam’sproduct,processandservicesolutionconceptsdevelopedbytheteams.ThepanelistsmetinpersonovertwodaysattheKelleySchoolofBusinessatIndianaUniversitytoreviewanddiscussthePowerPointpresentationscreatedbyeachteamaboutitsinnovation,pre-recordedvideosofeachteam’s5-minuteconceptpresentation,andpre-recordedvideosofeachteam’sresponsestoastandardizedsetofquestions.Aheadofthepanelmeeting(butafterthepanel’scompletionofworkontheassessmentrubric),thepanelistsalsoreceivedthebusinesscases(seeAppendixJ)createdbyeachoftheteamsfortheiradvancereview;panelistswereaskednottodiscussthesematerialswiththeircolleaguesaheadofthemeeting.Takentogether,theseelements(businesscase,PowerPointpresentations,videoof
AudienceViewpointsConsultingArtofScienceLearningResearchReport 26
conceptpresentation,andvideoofresponsestoquestions)formedthebasisforthepanel’soverallassessmentofeachteam’sinnovationoutcomes. Thereviewersdidnotknowwhichofthe16groupswereinthecontrolortreatmentcondition,andtheorderofteamswithineachcondition(i.e.,controlandtreatment)wasrandomized.SeeAppendixK.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 27
Limitations1.Numberofresearchsites:Initiallyitwasintendedtohavethreeresearchsites(Chicago,Worcester,andSanDiego),andhaveeachsiteruna5-weekInnovationChallenge.Unlikeinthefinaldesign,eachsitewouldhaveacohortofhighschoolstudentsandofearlySTEMcareerprofessionals,thoughthesetwogroupswouldnotbemixed.Forexample,thehighschoolcontrolgroupwouldmeetSaturdaymorning,theearlySTEMcareergroupSaturdayafternoon,thehighschooltreatmentgroupSundaymorning,andtheearlySTEMcareertreatmentgroupSundayafternoon.RecruitmentfortheChicagoincubatorwaslow,especiallyamongSTEMprofessionals,andwewereconcernedthatthegroupsrecruitedwouldnotresultinsamplesizeslargeenoughtoanalyze.TheChicagoInnovationChallengewasindefinitelypostponed,andwemadecorrespondingchangestothesamplemake-upinWorcesterandSanDiego.Thechangesallowedtheteamtoreducethesamplesize,asweonlyhadonepopulationtypepersite.Whilethisconflatesthesitewiththepopulationtype,itallowedrecruitmenttobesuccessfulwithinthosesites.2.Datacollectionlocations/needforlocaldatacollectors:DuetothetimeandfinancialfactorsofhavingmultipleAVCemployeestravelingfor10weekends,AVCrecruitedlocaldatacollectorsinWorcester,Massachusetts,andSanDiego,CaliforniatocollectobservationdataduringtheInnovationChallenge.AVCfoundlocaldatacollectorswhocamewithgoodrecommendationsfrompeersinthefieldandwhowereexperiencedinobservingbehaviorsininformallearningenvironments.AnAVCteammembertrainedtwolocaldatacollectorsduringthefirsttwoweeksinSanDiegoandthefirstthreeweeksinWorcesteronusingtheobservationandcollaborationratinginstrumentsandansweringbasicquestionsabouttheflowoftheworkday.AnAVCteammemberwasavailablebyphoneandtextmessageduringthelastweeksoftheinterventionincaseofemergencyorneedforclarification.Therefore,includingthreeAVCteammemberswhocollectedobservationdata,therewereatotalofsevendatacollectorsforoneinstrument.Itispossiblethathavingmoredatacollectorscouldintroducemoreerrorintothedataduetodifferentdefinitionsandinterpretationsofbehaviors.Tomitigatediscrepanciesalldatacollectorswereencouragedtocomparenotesthroughoutthedayanddiscussquestionsaboutwhethercertaingesturesorlanguageusedbyparticipantsdidindeedqualifyasspecificbehaviorsaccordingtotheinstrument.3.Placingpeoplepurposefullyintogroups:Puttingpeopleintothecontrolandtreatmentgroups,whilebeingbeneficialforspreadingthetwoconstructs(artandscience)out,mayhavehadsomeresidualeffectonthefindings.Amainpurposeofthestudywastocomparethetreatmentandcontrolgroupswithineachofthetwocohorts(highschoolstudents,earlycareerSTEMprofessionals),toseeifbeingexposedtothearts-basedapproachmadeadifferenceintheoutcomes.Giventhatcomparison,itwasimportanttocounterbalancethecontrolandtreatmentgroupssothatthecharacteristicsofeachgroupdidnotskewinacertaindirectionthatcouldimpacttheresults.Asresearchers,itwasalsounclearhowdiversethegroupswouldbe;iftheywereverydiversethenitwouldbeimportanttospreadthatdiversityevenlyacrossthecontrolandtreatmentgroups,andthenalsowithinthefourgroupsineachofthesecohorts.Studentswerecounterbalancedbyaccountingforan“artscore,”a“sciencescore,”anda“creativityscore;”theaveragescoresforeachwerematchedascloselyaspossible,sothesethreemainscoresmainlydrovewhereindividualswereplaced(whichcohort,thenwhichgroupwithineachcohort).Whilethisservedthepurposeofcounterbalancingtheeffectofthesevariablesontheresults,theremayhavebeensomeresidualeffecttomakingsurethegroupswerespreadevenlyalongthese
AudienceViewpointsConsultingArtofScienceLearningResearchReport 28
variables.Saidanotherway,theremayhavebeenanintangiblebenefittohavinggroupsbemoresimilartoeachotherthataffectedtheresults.4.Attritionduringthedatacollection:TherewasastrictnoabsencepolicyforparticipationintheInnovationChallengeworkshop.Anyparticipantwhomissedoneweekoftheworkshopwasnotallowedtoreturnforanyfollowingworkshopday.Theprogramfacilitatorsfeltthatparticipantswhomissedaworkshopdaywouldfalltoofarbehindincontentunderstandingandgroupcollaborationandcouldnotbereintegratedtothegroup.ItispossiblethatparticipantswhofelttheInnovationChallengewasnotmeetingtheirexpectationsdecidedtodropoutoftheprogram.ThoseindividualswhodidnotperceivetheInnovationChallengetobebeneficialforthemwouldhavelikelyshowednochangeintheirbehaviorsandskillsthroughoutallmethodsofdatacollection,PreSurveytoPostSurvey,CollaborationRatings,Observations,etc.Thiscouldpossiblyskewthedataasweonlyhaveresponsesandratesofchangeforparticipantswhocompletedtheentireintervention.Thisisaconsequenceofthenatureoftheinterventiondesignedasaworkshop,notasamandatoryexperimentaldesignwhereparticipantsmustcompleteallworkshopdays.5.Usingadolescentcurriculumforbothgroups.InordertobeconsistentandallowforacomparisonbetweenthehighschoolstudentsandadultswhowereatanearlystageintheirSTEMcareer,thesamecurricularapproachwasusedforbothcohorts.Thecurriculumusedinthisstudywasspecificallydesignedforusewithadolescents,andwhiletherewasthepossibilitythatemployingastudent-focusedcurriculumcouldbeexperienceddifferentlybytheearlycareerSTEMprofessionalsitwasdeemednecessarybytheresearchersinordertocomparetheutilityoftheapproachwithbothcohorts.ItispossiblethatsomeofthelackoffindingsamongtheearlySTEMprofessionalsmayresultfromthecurriculumhavingbeendevelopedmorefortheadolescents.Furtherstudytodeterminewhetheradultfindingswouldchangewiththesubstitutionofacurriculumspecificallydesignedforusewithadultswouldbevaluable.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 29
SamplingandCharacteristicsoftheSampleIntheoriginalplansfortheInnovationChallengeworkshopseachlocationwouldhaveacohortofhighschoolstudentparticipantsandacohortofearlycareerSTEMprofessionalparticipants.BecauseofdifficultiesinrecruitmentofearlycareerSTEMprofessionalparticipants,inWorcestertheprojectdirectorsandAVCdecidedtocanceltheadultworkshopinWorcesterandrunonlythehighschoolstudentprogram,thenrunonlytheearlycareerSTEMprofessionalsprograminSanDiego.
HighSchoolStudentsTheArtofScienceLearningteamatthehostsitetheEcoTariuminWorcester,ledbyAoSL’sWorcesterIncubatorDirectorJoyceKressler,conductedrecruitmentofparticipants,reachingouttocontactsinthecommunitytoraiseawarenessabouttheworkshop.Projectcommunitypartnersincludedregionalpublicandprivateschools,nonprofitorganizations,collegesanduniversities,informallearningcenters,parentorganizations,businessesandeducators.Altogether,AoSLcontactedmorethan15highschools(forexample,WorcesterTechnicalHighSchoolandDohertyHighSchool),morethan10communitycollegesanduniversities(forexample,WorcesterStateUniversityandUniversityofMassachusettsMedicalSchool),6non-profitorganizations(forexample,theBoysandGirlsClubofAmerica),7STEM-relatedcompanies(forexample,Intel),andmorethan10relevantprofessionalorganizations(forexample,MassachusettsBiomedicalInitiatives).Informationandinvitationstoregisterfortheworkshopweresentviaemailtoacontactateachschoolorinstitution,andtheAoSLteamattheEcoTariumalsomadephonecallstocertainpartnerswhereapplicable.Additionally,theEcoTariumpromotedtheWorcesterInnovationChallengethroughapostingontheirFacebookpagethatwaspaidforbyfundsfromtheArtofScienceLearninggrant.ProspectiveparticipantsintheInnovationChallengeworkshopwereaskedtofilloutashortquestionnaireabouttheirexperiencewithSTEM(Science,Technology,Engineering,Math)subjectsandtheircurrentinvolvementwithcreativeorartisticpursuits.RecruitersfocusedtheireffortstofindparticipantswhohadsomelevelofinterestorparticipationintheSTEMfields,butitwasnotnecessarythataparticipantbeactivelyinvolvedinanycreativeorartisticendeavor.Thequestionnairealsocollectedinformationaboutthegradelevelandhighschooltheparticipantattended,informationthatwouldbeusedtogroupparticipantsintoteamssothatpeoplewhoalreadykneweachotherwouldnotbeonthesameteams.Recruitmentfocusedonhighschoolstudentsin10thand11thgrades;however,studentsin9thand12thgradeswereacceptedtotheWorcesterInnovationChallenge.ParticipantswerealsorequiredtosubmittwoelectronicconsentformsinordertoparticipateintheWorcesterInnovationChallenge:aformconsentingforthemselves,andasecondformwiththeassentoftheirparentorguardian.ParticipantswhodidnotsubmitthesetwoseparateformswerenotallowedtocompletethePreWorkshopSurveyperIRBregulationsandthereforecouldnotcontinueintherecruitmentprocess.BytheendoftherecruitmentperiodthereweremorestudentswhohadregisteredfortheWorcesterInnovationChallengeworkshopthancouldparticipate;atotalof190applicationswerereceived,someincomplete,andonly88participantscouldbeacceptedtotheprogram.Participantsweretakenonafirstcomefirstservedbasis,thendividedbytheirschoolsandgradeintoeighttemporaryteamsofelevenstudentswiththeunderstandingthatafewstudentsmightnotshowthefirstday,orwoulddropoutafterthefirstworkshopsession.Thetemporaryteamswereconfiguredinawaythatnoteamwouldhavemorethanoneparticipantfromthesameschoolandgrade;however,whileteamsdidhavemore
AudienceViewpointsConsultingArtofScienceLearningResearchReport 30
thanoneparticipantfromthesameschool,theywerefromdifferentgrades.Considerationwasalsogiventoparticipantswhohadrequestedparticipatingineitherthemorningorafternoonworkshopsduetoschedulingortransportationconflicts.Otherthanthesestipulations,participantswererandomlyassignedtothecontrolgroup(afternoonsession)ortreatmentgroup(morningsession).ParticipantswhoattendedaschoolwithalreadyconsiderablerepresentationontheteamswerecontactedandinformedthattheywouldbeunabletoparticipateintheWorcesterInnovationChallenge.ParticipantswerethensortedbasedontheirresponsestothePreWorkshopsurveyabouttheirexperiencewithartandscienceeducationandtheir“creativityscore”(acompositescoreofspecificquestionsfromthePreWorkshopSurvey).Teamswereconfiguredtohaveverysimilaroverallaveragesoftheirscoresforart,scienceandcreativity,withoutanyoverlappingofhighschoolandgradelevel.Itwascriticaltomaintaintheseaveragescores,whichdidnothaveastatisticallysignificantdifference,inorderfortheteamstobecomparablebetweencontrolandtreatmentgroups.Onthefirstdayoftheworkshopparticipantswereresortedintogroupsifindividualsonthesameteamhadpersonalrelationshipswitheachother,whichcouldnothavebeenforeseenbytheplanningteam.Teamswerealsoresortedintheeventthatmultipleparticipantstemporarilyplacedonacertainteamdecidednottoparticipate,toevenouttheaveragesizeofeachteam.However,onceindividualswereofficiallyplacedonateamduringthefirsthouroftheWorcesterInnovationChallenge,individualswerenotallowedtoswitchtoadifferentteamatanypointduringthelengthoftheChallenge.TheInnovationChallengeworkshopsforthehighschoolstudentcohortwereheldonconsecutiveSundaysfromOctober26th,2014throughNovember23rd,2014.Eachhalfdaysessionranforaperiodoffourhours;themorningsessionwasheldfrom9:00amto1:00pmandtheafternoonsessionwasheldfrom1:30pmto5:30pm.ParticipantsintheWorcesterworkshopwereofferedastipendof$250,whichtheywouldreceiveafterattendingallfiveworkshopdatesandcompletingthepostworkshopsurvey.TheworkshopswereheldattheEcoTarium,asciencemuseumandnaturecenterinWorcester,Massachusetts.ByweekfiveoftheWorcesterInnovationChallengeatotalof65individualsparticipatedinthecohort.ThismeansthattherewassomeattritionthroughoutthefiveweeksoftheChallenge;however,themajorityofindividualswhodroppedoutoftheprogramdidsobetweenweekoneandweektwo.Asnotedabove,theattritionratewasconsiderablyhigherforthecontrolgroupthanforthetreatmentgroup.Thetreatmentgroupsaw32ofthe34initialparticipantscompletethefivesessions,withanattritionrateof6%;whilethecontrolgroupsaw33oftheinitial38participantscompletethefivesessions,withanattritionrateof13%.Thereweremorefemaleparticipants(68%)thanmaleparticipants(32%),withallparticipantsintheWorcesterInnovationChallengebetweentheagesof15and18yearsold(seeTable3).Almostalloftheparticipantswerefull-timestudents(95%)andcamefrommorethan20differenthighschoolsinthearea.Themajorityofparticipantswerenotemployed(85%),thoughsomewereemployedpart-time(12%).ThelargestsinglegroupofparticipantsreportedlyidentifiedwiththeCaucasianorWhiteethniccategory(41%),withthenextmostcommonlyidentifiedcategoriesbeingAsian(15%)andAfricanAmericanorBlack(14%).AsmallerpercentageofparticipantsidentifiedasHispanicorLatino(8%)andAmericanIndianorAlaskaNative(1%).Someparticipantspreferrednottoanswer(11%)andsomechose“Other”andwroteinhowtheyidentifiedthemselves.Forexample,someparticipantswrote:
NativeAmerica,Portuguese,AfricanAmerican,FrenchCanadian,etc.Arabic
AudienceViewpointsConsultingArtofScienceLearningResearchReport 31
Indian EuropeanandAfrican IamhalfCaucasianandhalfPakistani
Table3:DemographicsforHighSchoolStudentParticipantsCharacteristic Control(n=33) Treatment(n=32) Total(n=65)Gender
Male 27% 38% 32%Female 73% 62% 68%
AgeCategory 15to18 100% 100% 100%
EthnicCategory CaucasianorWhite 42% 42% 41%AfricanAmericanorBlack 21% 6% 14%Asian 6% 25% 15%HispanicorLatino 6% 9% 8%AmericanIndianorAlaskaNative 0% 3% 1%Other 9% 12% 11%Prefernottoanswer 15% 3% 9%
CurrentLevelofEducation Stillenrolledinhighschool 85% 97% 91%GED 3% 0% 1%Didnotanswer 12% 3% 8%
StudentStatus Part-timestudent 0% 6% 3%Full-timestudent 97% 94% 95%Didnotanswer 3% 0% 1%
GradeLevel 9th 6% 0% 3%10th 48% 44% 46%11th 39% 31% 35%12th 6% 25% 15%
EmploymentStatus Notemployed 88% 81% 85%Employedpart-time 9% 16% 12%Didnotanswer 3% 3% 3%
(Table3continuedbelow)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 32
Table3:DemographicsforHighSchoolStudentParticipantscontinuedCharacteristic Control(n=33) Treatment(n=32) Total(n=65)SchoolAttendance
WorcesterTechnicalHighSchool 18% 19% 18%UniversityParkCampusSchool 18% 3% 10%DohertyMemorialHighSchool 3% 16% 9%NorthHighSchool 6% 9% 8%BancroftSchool 3% 12% 8%SouthHighCommunitySchool 9% 3% 6%BurncoatHighSchool 9% 0% 5%WachusettRegionalHighSchool 3% 6% 5%ShrewsburyHighSchool 6% 0% 3%MassachusettsAcademyofMathandScience 3% 3% 3%LincolnSudburyHighSchool 6% 0% 3%PlainfieldHighSchool 0% 6% 3%HopkintonHighSchool 3% 3% 3%AdvancedMathandScienceAcademyCharter 0% 6% 3%WorcesterAcademy 0% 6% 3%SaintJohn’sHighSchool 0% 3% 1%HopedaleJunior-SeniorHighSchool 3% 0% 1%QuinebaugMiddleCollege 3% 0% 1%NorthamptonHighSchool 0% 3% 1%HolyNameJuniorSeniorCentralCatholic 3% 0% 1%AuburnHighSchool 3% 0% 1%
EarlyCareerSTEMProfessionalsDr.NanRenner,DirectoroftheArtofScienceLearningSanDiegoIncubatorandtheAoSLteamconductedrecruitingfortheSanDiegoInnovationChallenge.Theteamsreachedouttocontactsinthecommunitytoraiseawarenessabouttheworkshop,suchasnonprofitorganizations,educatorsandothereducationalinstitutions.InformationabouttheSanDiegoInnovationChallengewasdistributedviatheBalboaParkLearningInstitutee-newsletter,whichhas1,800subscribers.TheteaminSanDiegoemailedapproximately150educatorsfromSanDiegoStateUniversity,UniversityofCaliforniaSanDiego,andthecommunitycollegesinthearea.Additionally,theyreachedouttoabout50educatorsfromBalboaParkandcontactedmorethan100additionalindividualsfromtheirpersonalnetworks.SpecificcriteriaforselectionincludedcurrentemploymentinaSTEM-relatedprofessionalfieldorrole(includingSTEMeducation),lessthansevenyearsofprofessionalexperienceworkinginaSTEMfield,completionofatleasttwoyearsofcollegeeducationandprofessionalcompetenceinEnglish.TwoAVCresearchersindependentlyreviewedtheinformationsubmittedinthequestionnairetodetermineeligibilityforeachpotentialparticipant.Eachpotentialparticipantwaslabeledas“yes,”“no,”or“maybe”basedonthecriteria.Inordertoprovideasdiverseandbalancedagroupaspossible,secondarycriteriawerealsotakenintoaccount.Theseincludedage,whetherornotpotentialparticipantswereteachersorcurrentstudents,andartsbackground.Alsotakenintoconsiderationwasavailability;participantsunabletocommittoattendingallofthesessionswerenotinvitedto
AudienceViewpointsConsultingArtofScienceLearningResearchReport 33
participate.Aftereachpotentialparticipantwascategorizedindependently,thetworesearchersthencomparedandfinalizedthelistofthoseinvitedtoparticipate.ParticipantsfortheInnovationChallengeworkshopwereaskedtofilloutashortquestionnaireabouttheirexperiencewithSTEMsubjects(Science,Technology,Engineering,Math)andtheircurrentinvolvementwithcreativeorartisticpursuits.RecruitersfocusedtheireffortstofindparticipantswhowereintheearlystagesoftheircareerintheSTEMfield,howeversomeparticipantswhowereacceptedtotheprogramdidnotworkdirectlyintheSTEMfield,andsomeparticipantsdidinfactworkwithintheSTEMfieldbutwerenotinthebeginningoftheircareer.Itwasnotnecessarythataparticipantparticipateinanycreativeorartisticendeavor.Thequestionnairealsoaskedwhetherparticipantswerestudents,employedpart-timeorfull-time,orunemployed/retired.ParticipantswererequiredtosubmitanelectronicconsentforminordertoparticipateintheSanDiegoInnovationChallenge.ParticipantswhodidnotsubmitthisformwerenotallowedtocompletethePreWorkshopSurveyperIRBregulationsandthereforecouldnotcontinueintherecruitmentprocess.TherewerefewerobstaclesforincludingandsortingteamsintheSanDiegoInnovationChallengebecausetheparticipantswerenotminors,andthereforeonlyhadtocompleteoneconsentform.BytheendoftherecruitmentperiodthereweremoreindividualswhohadregisteredfortheSanDiegoInnovationChallengeworkshopthancouldparticipate;atotalof175applicationswerereceived,someincomplete,andonly88participantscouldbeacceptedtotheprogram.Participantsweretakenonafirstcomefirstservedbasis,thendividedintoeighttemporaryteamsofelevenindividualswiththeunderstandingthatafewindividualsmightnotshowthefirstday.Considerationwasalsogiventoparticipantswhohadrequestedparticipatingineitherthemorningorafternoonworkshopsduetoschedulingortransportationconflicts.Otherthanthesestipulations,participantswererandomlyassignedtothecontrolgroup(afternoonsession)andtreatmentgroup(morningsession).ParticipantswerethensortedbasedontheirresponsestothePreWorkshopsurveyabouttheirexperiencewithartandscienceeducationandtheir“creativityscore”(acompositescoreofspecificquestionsfromthePreWorkshopSurvey).Teamswereconfiguredtohaveanoverallaverageofartscore,sciencescoreandcreativityscore.Itwascriticaltomaintaintheseaveragescores,whichdidnothaveastatisticallysignificantdifference,inorderfortheteamstobecomparablebetweencontrolandtreatmentgroups.Onthefirstdayoftheworkshopparticipantswereresortedintogroupsifindividualsonthesameteamhadpersonalrelationshipswitheachother,whichcouldnothavebeenforeseenbytheplanningteam.Teamswerealsoresortedintheeventthatmultipleparticipantstemporarilyplacedonacertainteamdecidednottoparticipate,toevenouttheaveragesizeofeachteam.However,onceindividualswereofficiallyplacedonateamduringweekoneoftheWorcesterInnovationChallenge,individualswerenotallowedtoswitchtoadifferentteamatanypointduringthelengthoftheChallenge.TheInnovationChallengeworkshopsfortheearlycareerSTEMprofessionalcohortwereheldonconsecutiveSaturdaysfromJanuary10th,2015throughFebruary7th,2015.TheworkshopswereheldatBalboaParkinSanDiego,California.Eachhalfdaysessionranforaperiodoffourhours;themorningsessionwasheldfrom9:00amto1:00pmandtheafternoonsessionwasheldfrom1:30pmto5:30pm.ParticipantsintheWorcesterworkshopwereofferedastipendof$500,whichtheywouldreceiveafterattendingallfiveworkshopdatesandcompletingthepostworkshopsurvey.ThestipendfortheSan
AudienceViewpointsConsultingArtofScienceLearningResearchReport 34
DiegoparticipantswassignificantlylargerthanthestipendfortheWorcesterparticipants-thiswasapurposefuldecisionmadetogiveadultsagreaterincentivetocontinuetotheendoftheprogram.ByweekfiveoftheSanDiegoInnovationChallengeatotalof69individualsparticipatedinthecohort.ThismeansthattherewassomeattritionthroughoutthefiveweeksoftheChallenge;however,themajorityofindividualsdroppedoutoftheprogramweekone.Asnotedabove,theattritionratewashigherforthecontrolgroupthanforthetreatmentgroup.Researchersnotedthat37ofthe40initialparticipantsinthetreatmentgroupcompletedallfivesessions,withanattritionrateof8%;whilethecontrolgroupsaw32oftheinitial36participantscompletethefivesessions,withanattritionrateof11%.DuetothenatureoftherecruitingrequirementsfortheSanDiegoInnovationChallenge,therewereawidervarietyofagesinthisprogram,comparedtoWorcester(seeTable4).Themajorityofparticipantswereyoungadults19to25yearsold(42%)or26-30yearsold(17%)inthebeginningstagesoftheircareer.Thereweresmallerpercentagesofadults,31to40yearsold(26%),41to50yearsold(7%),51to60yearsold(6%)and61to70yearsold(1%).TherewerestillmorewomenparticipantsintheSanDiegoChallenge,likeWorcester;however,theratioofwomentomenwasmoreeven(59%women,41%men).ThemajorityofparticipantsreportedlyidentifiedwiththeCaucasianorWhiteethniccategory(52%),withthenextmostcommonlyidentifiedcategoriesbeingHispanicorLatino(19%)andAsian(16%).AsmallerpercentageofparticipantsidentifiedasAfricanAmericanorBlack(4%)andNativeHawaiianorotherPacificIslander(1%).Someparticipantspreferrednottoanswer(7%)andsomechose“Other”andwroteinhowtheyidentifiedthemselves.Forexample,someparticipantswroteinamoredetaileddescriptionoftheirethniccategorysuchas:
Chicano Brazilian AlsoHispanic,wouldnotletmecheckmorethanone EuropeanThemajorityofparticipantswerenotcurrentlyattendingaschool(62%)butthereweresomeparticipantswhowerefull-timestudents(29%)orpart-timestudents(9%).OfthoseindividualswhoweretakingclassesatthetimeoftheSanDiegoInnovationChallenge16%wereenrolledincollegeorcommunitycollegecourses,and22%wereenrolledingraduateschool.ThoseparticipantswhowerestudentsatthetimeoftheworkshopattendedavarietyofschoolsintheSanDiegoarea,includingUniversityofCalifornia,SanDiego(17%)andSanDiegoStateUniversity(7%).Manyparticipantshadalreadycompletedacommunitycollegeortechnicalcertificate(9%),acollegedegree(43%)oragraduateorpostgraduatedegree(36%).Asmallpercentageofparticipants(10%)reportedtheirhighestlevelofeducationasahighschooldiplomaorGED.AlmostallparticipantswereinvolvedincareersthatwereSTEMrelated(77%)orinsomewayrelatedtotheSTEMfield(20%).Themajorityofparticipantswereemployedeitherfull-time(56%)orpart-time(26%)withjustafewparticipantsreportingtobeunemployedorretired(4%)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 35
Table4:DemographicsforEarlyCareerSTEMProfessionalParticipantsCharacteristic Control(n=32) Treatment(n=37) Total(n=69)Gender
Male 37% 43% 41%Female 62% 57% 59%
AgeCategory 19to25 44% 40% 42%26to30 16% 19% 17%31to40 22% 30% 26%41to50 6% 8% 7%51to60 12% 0% 6%61to70 0% 3% 1%
EthnicCategory CaucasianorWhite 47% 57% 52%AfricanAmericanorBlack 3% 5% 4%Asian 22% 11% 16%HispanicorLatino 12% 24% 19%NativeHawaiianorotherPacificIslander 3% 0% 1%Prefernottoanswer 12% 3% 7%
StudentStatus Notcurrentlyastudent 62% 62% 62%Part-timestudent 12% 5% 9%Full-timestudent 25% 32% 29%
CurrentEducationLevel Collegeorcommunitycollege 22% 11% 16%Graduateschool 16% 27% 22%Notapplicable 62% 62% 62%
HighestLevelofEducation Highschool/GED 12% 8% 10%CommunityCollege/technicaltraining 6% 11% 9%CollegeDegree(BA/BS) 47% 40% 43%GraduateorPostgraduateDegree 31% 40% 36%Didnotanswer 3% 0% 1%
CurrentSTEMProfessional? Yes 75% 78% 77%No 0% 3% 1%Tosomeextent 25% 16% 20%I’mnotsure 0% 3% 1%
EmploymentStatus Notemployedand/orretired 9% 16% 13%Employedpart-time 28% 24% 26%Employedfull-time 59% 54% 56%Didnotanswer 3% 5% 4%
(Table4continuedbelow)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 36
Table4:DemographicsforEarlyCareerSTEMProfessionalParticipantscontinuedCharacteristic Control(n=32) Treatment(n=37) Total(n=69)SchoolAttendance
UniversityofCalifornia,SanDiego 12% 22% 17%SanDiegoStateUniversity 9% 5% 7%PointLomaNazareneUniversity 3% 3% 3%SouthwesternCollege 0% 3% 1%MiamiUniversity,Ohio 0% 3% 1%NationalUniversity 3% 0% 1%CaliforniaStateUniversity,LongBeach 0% 3% 1%SanDiegoCityCollege 3% 0% 1%CUNYGraduateCenter,NewYork,NY 3% 0% 1%SanDiegoStateUniversity&UniversityofCalifornia,Davisjointprogram 3% 0% 1%
AudienceViewpointsConsultingArtofScienceLearningResearchReport 37
Hypothesis1Findings:CreativeThinkingSkillsFundamentally,theresearchcomponentoftheArtofScienceLearningprojectwasdesignedtoinvestigatewhetherparticipatinginarts-basedinnovationtraininggivesoneanadvantageoverparticipatinginmoretraditionalinnovationtrainingofthetypethatonemightencounterinanacademicorworkplacesetting.Withinthisframework,ourresearchwasbrokendownintothreeprimaryhypotheses.Thefirstoneconsideredtherelativeadvantageofarts-basedinnovationtrainingonarangeofcreativethinkingskills.Theteambrokethishypothesisdownintoaseriesofsub-hypothesesinordertobetterinvestigateeachelement.Hypothesis1:Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,improvesanindividual'screativethinkingskillsincludingcriticalthinking,divergentthinking,problemidentification,convergentthinkingandproblemsolving.
• Hypothesis1a:Artsbasedinnovationtrainingincreasesanindividual'sabilitytoemploydivergentthinkingovertraditionalinnovationtraining.
• Hypothesis1b:Artsbasedinnovationtrainingincreasesanindividual'sabilitytoemployconvergentthinkingovertraditionalinnovationtraining
• Hypothesis1c:Artsbasedlearninginfluencesindividuals’criticalthinkingskills.Asmentionedwithinthemethodssection,eachofthesesub-hypotheseswasmeasuredthroughavarietyofmethods.Detailedinformationontheunderlyingconstructsofeachofthesescalescanbefoundinthesummarytableofmethods(seeTable2).OneoftheareaswherewefoundthemostsignificantdifferencesbetweengroupsandbetweencohortswaswithintheCreativeSkillsmeasure.Participantswereaskedtocompleteabrief“InnovationWarm-up”exerciseatthestartofthefirstdayoftheworkshop,andasimilar“InnovationWrap-up”atthecloseofthefinaldayoftheworkshop.ParticipantswereaskedtoidentifyproblemsrelatedtoagivenInnovationChallenge,selectonetoworkon,generatepossiblesolutionstotheselectedproblem,selectonesolution,andexplaintheirchoices(seeAppendicesFthroughHforthefullexercises).AdifferentInnovationChallengewasusedfortheseconduseoftheexercisetopreventanypracticeeffects.Participantshad15minutestocompletetheskillstest.ThesetaskswerederivedfromconceptssimilartothechallengesnotedinRuncoandBasadur’s(1993)articleonAssessingIdeationalandEvaluativeSkillsandCreativeStylesandAttitudes.Arubricthatincludedtheskillcategorieslistedbelowwascreatedforthescoringofthistest.Therubricwasdesignedtoinvestigatethefollowingcreativethinkingskills,andincludethetypeofskillitexamined:
Skill1:Howmanydistinctproblemswereidentified?(Divergent)Skill2:Howmanyideaclustersdothoseproblemsrepresent?(Divergent)Skill3:Howclearistheproblemstatementasrelatedtothechallenge?(Convergent)Skill4:Howmanyreasonsgivenforwhytheindividualchoosesthatproblem?(Convergent)Skill5:Howmanydistinctsolutionsweregenerated?(Divergent)Skill6:Howstrongisthesolutionstatement?(Convergent)Skill7:Howmanyreasons/evidencestatementsgivenforselectionoftheproblem?
(Convergent)Skill8:Howmanyreasons/rationalesgivenforchoosingsolution?(Convergent)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 38
Skill9:Howspecificiswhattheparticipantproposestodo?(Convergent)Skill10:Howspecificastohowtheparticipantproposestoenacttheirsolution?(Convergent)Skill11:Howmanyspecificideaclustersdothesesolutionsrepresent?(Divergent)
DifferencesWithingroupsOurcoreobjectivewastodeterminewhetherornotthetypeofinnovationtrainingindividualsreceivedimpactedtheircreativethinkingskills.Beforewecouldcomparetheimpactoftrainingbetweenthecontrolandtreatmentgroups,weneededtocomparetheimpactofthetrainingoneachgroupindividually.AsshowninTable5below,thehighschoolstudenttreatmentgroupshowedstatisticallysignificantincreasesfrompretesttoposttestwithin6ofthe11creativethinkingskillvariables.Forthecontrolgroup,therewerenogainsonanyvariableafterthetraining;infact,thehighschoolcontrolgrouphadhigherpretestscoreswithinthecriticalthinkingvariablesand3ofthecreativethinkingskillswhencomparedwiththeirpost-testscores.Astrikingpatternemergedforthehighschoolstudents:outofthe13totalvariablesthatshowedsignificantdifferences,onlyoneofthetwogroups(controlortreatment)showedasignificantdifference.Furthermore,12ofthese13variablesshowedapositiveresultinthedirectionofthetreatmentgroup.Forexample,thehighschoolstudenttreatmenthad7variablesthatshowedasignificantincreasefrompretopost,andnoneoftheseweresignificantlydifferentforthecontrolgroup;treatmentincreased,controlstayedthesame.Likewise,thehighschoolcontrolgrouphad5variablesthatshowedasignificantdecreasefrompretopostforthehighschoolcontrolgroup,yettherewasnosignificantdifferenceforthetreatmentgroup;treatmentstayedthesame,whilecontroldecreased.Theseresultsstronglysuggestthatinadditiontoanybeneficialimpactofarts-basedlearningonspecificcreativethinkingskills,arts-basedlearningmayhavethecapacitytoovercomeandneutralizeanynegativeimpactoftraditionalinnovationtrainingonhighschoolstudentcreativethinking.Itwillbeinterestingtofurtherinvestigatethispossibleeffect,aswellastoconsiderwhetherthereareanypositiveimpactsoftraditionalinnovationtrainingonotherareaswedidnotmeasure,suchasapotentialincreaseincontentknowledgeaboutinnovation.Thispossibleimpactmeritsfurtherinvestigation.Itwouldalsobeinterestingtoconsiderwhetherthereareanypositiveimpactsofeitherarts-basedortraditionalinnovationtrainingonotherareaswedidnotmeasure,suchasapotentialincreaseincontentknowledgeaboutinnovation.Thehighschooltreatmentgroupshowednoperceivablegainonfourofthecreativethinkingskillsoronthecriticalthinkingandcreativityself-reportscales;theincreasewasonlymeasuredonthecreativethinkingskillstest,whichsuggeststhepossibilitythatarts-basedtrainingspecificallyhasanimpactonthetypeofskillsthistestwasdesignedtogauge:theabilitytoidentifyaproblem,articulatepotentialsolutions,andidentifyasolutionandarticulatearationaleforthatsolution.Thebreadthandspecificityofproblemsandsolutionsdescribedarealsoevaluatedwithinthisassessment.Itisalsopossiblethatdespiteeffortstobalancethegroupswithregardtoindependentvariablessuchaspriorartsexperience,scienceexperience,andcreativityscores,thecontrolgroupenteredwithahigherlevelofcreativeskills,andthusshowednochange;orthatthecreativeskillsassessment,withascoringrubricdevelopedaftertheimplementationoftheassessment,isinsomeunknownwaybiased.Nonetheless,thesimplestexplanationisthatthearts-basedtrainingincreasesthecreativethinkingskillsofhighschoolstudents.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 39
TheresultsfromtheearlycareerSTEMprofessionalsarelessdefined.Asinthehighschooltreatmentgroup,buttoalesserdegree,theearlycareerSTEMprofessionalstreatmentgroupshowssomeincreasesincreativethinkingskills.Theyshownochangeinthecriticalthinkingskillsmeasures.Ononeself-reportscale,Creativity1,theearlycareerSTEMprofessionalstreatmentgrouphadstatisticallysignificantlyhigherscoresonthepre-test.Aswedidnotexpectthemtolosecreativityoverthecourseofthetraining,itisunclearwhatthisfindingmeans.TheearlycareerSTEMprofessionalscontrolgroupshowedlittleimpactfromthetraining,withtheexceptionofincreasesintwocreativeskills:numberofsolutionstatementsgenerating,andnumberofideaclustersrepresentedbythesesolutions.Table5:SummaryTableofIndividualCreativeThinkingSkillsScores(includingCriticalThinking)
Note:Seepages37-38aboveforexplanationofSkill1throughSkill11.
DifferencesBetweengroupsThissectionwillfocusonthedifferencesbetweenthecontrolandtreatmentgroups,organizedwithinthesub-hypothesesarticulatedabove.ThefourtablesbelowaddressthemainHypothesis1(seeTable6)aswellassub-hypotheses1a(seeTable7),1b(seeTable8),and1c(seeTable9).Forthehighschoolstudentcohorts,thesetablesshowedalargenumberofsignificantdifferencesthatwerefoundincreativeandcriticalthinkingscoresbetweenthecontrolandthetreatmentgroups,withthetreatmentgroupsoutperformingthecontrolgroups,mostfrequentlywithinthecreativeskillstests.Innocasedidcontroloutperformtreatment.WithrespecttoSub-Hypothesis1a(divergentthinking,Table7),thetreatmentgroupsignificantlyoutperformedthecontrolgroupin4of5variablesmeasured,withtheremainingvariableshowingnosignificantdifference.WithrespecttoSub-Hypothesis1b(convergentthinking,Table8),thetreatmentgroupsignificantlyoutperformedthecontrolgroupin3of6variablesmeasured,withtheremaining3variablesshowingnosignificantdifference.WithrespecttoSub-Hypothesis1c(criticalthinking,Table9),thetreatmentgroupsignificantlyoutperformedthecontrolgroup.
VariableHighSchool
StudentsTreatmentHighSchool
StudentsControlEarlyCareerSTEM
ProfessionalsTreatmentEarlyCareerSTEM
ProfessionalsControlCriticalThinking1 -- Pretesthigher -- --CriticalThinking2 -- Pretesthigher -- --
Mini-ECCI Posttesthigher
CPSP CreativeThinkingSkills:Skill1
Posttesthigher--
Posttesthigher--
Skill2 Posttesthigher -- -- --Skill3 Posttesthigher -- Posttesthigher --Skill4 -- Pretesthigher Posttesthigher --Skill5 Posttesthigher -- -- PosttesthigherSkill6 Posttesthigher -- -- --Skill7 -- Pretesthigher -- --Skill8 -- Pretesthigher -- --Skill9 Posttesthigher -- -- --Skill10 -- -- -- --Skill11 Pretesthigher -- -- Posttesthigher
AudienceViewpointsConsultingArtofScienceLearningResearchReport 40
ThesehighschoolfindingsmustbeinterpretedinthecontextofTable5above;insomecases,thetreatmentgroup’sstrongercomparativeperformanceresultedinpartfromadeclineintheperformanceofthecontrolgroupoverthecourseofthefivesessions,aswellasfromanincreaseintheperformanceofthetreatmentgroup.Inparticular,thecontrolgrouphadstatisticallyhigherscoresoncriticalthinkingandsomecreativeskillsonentrytothetrainingthantheyshowedfiveweekslater.SoinTable6throughTable9below,whilethehighschooltreatmentgroupsoutperformedthehighschoolcontrolgroupsbyastatisticallysignificantdifference,animportantfactorcontributingtothisdifferenceisthatinsomecases,thehighschooltreatmentgroupshadrelativelymodestpositivechangesincriticalthinkingskills,whilethehighschoolcontrolgroupshadadropinscoresfrompretesttoposttest.Whilethehighschooltreatmentgroupsdoshowincreasesincreativeskillsduringthecourseofthetraining,thedifferencesbetweentheirgroupandthehighschoolcontrolgroupisenhancedbythefactthatthehighschoolcontrolgrouphaddecreasesintheirscores,showinghigherscoresontheentrypretestthantheposttest.Onepossibleexplanationthatwarrantsfurtherstudyisthatamoretraditionalapproachtolearninginnovationmayinsomewaydepresscreativethinking,whilethearts-basedapproachmayoffsetthatnegativeimpact.SomeofthesedifferenceswerealsofoundwithintheearlycareerSTEMprofessionalsgroups,buttoaveryslightdegree(seeTable10throughTable13below).TherewerenostatisticallysignificantdifferencesbetweentheearlycareerSTEMprofessionalstreatmentandcontrolgroups,onlyaslightnon-significanttrendinCreativeSkill3(clarityoftheproblemstatement)towardsbetterperformanceinthetreatmentgroup. Table6:SummaryTableofDifferencesbetweenTreatmentandControlGroups,Hypothesis1 HighSchoolStudents EarlyCareerSTEMProfessionalsPretest-PosttestDifferenceScores
SignificantGroupDifferences
GroupwithBetterPerformance
SignificantGroupDifferences
GroupwithBetterPerformance
CreativeSkill2 Yes Treatment No -CreativeSkill6 Yes Treatment No -CreativeSkill7 Yes Treatment No -CreativeSkill8 Yes Treatment No -CreativeSkill9 No - No -CreativeSkills10 Trend Treatment No -CreativeSkills11 No - No -Mini-ECCI Yes Treatment No -Slover-LinnettCreativeProcess No - No -
Basadur’sProblemSolving(CPSP) No - No -
CommonPurpose No - No -ObservedCommonPurpose
No - No -
CreativeSkills11 No No -Note:Seepages37-38aboveforexplanationofSkill1throughSkill11.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 41
Table7:SummaryTableofDifferencesbetweenTreatmentandControlGroups,Hypothesis1a HighSchoolStudents EarlyCareerSTEMProfessionalsPretest-PosttestDifferenceScores
SignificantGroupDifferences
GroupwithBetterPerformance
SignificantGroupDifferences
GroupwithBetterPerformance
CreativeSkills1 No - No -CreativeSkills2 Yes Treatment No -CreativeSkills3 Yes Treatment Trend TreatmentCreativeSkills4 Yes Treatment No -CreativeSkills5 Yes Treatment No -Note:Seepages37-38aboveforexplanationofSkill1throughSkill11. Table8:SummaryTableofDifferencesbetweenTreatmentandControlGroups,Hypothesis1b HighSchoolStudents EarlyCareerSTEMProfessionalsPretest-PosttestDifferenceScores
SignificantGroupDifferences
GroupwithBetterPerformance
SignificantGroupDifferences
GroupwithBetterPerformance
CreativeSkills6 Yes Treatment No -CreativeSkills7 Yes Treatment No -CreativeSkills8 Yes Treatment No -CreativeSkills9 No - No -CreativeSkills10 Trend Treatment No -CreativeSkills11 No - No -Note:Seepages37-38aboveforexplanationofSkill1throughSkill11. Table9:SummaryTableofDifferencesbetweenTreatmentandControlGroups,Hypothesis1c HighSchoolStudents EarlyCareerSTEMProfessionalsPretest-PosttestDifferenceScores
SignificantGroupDifferences
GroupwithBetterPerformance
SignificantGroupDifferences
GroupwithBetterPerformance
CriticalThinking Yes Treatment No -
AudienceViewpointsConsultingArtofScienceLearningResearchReport 42
Inthefollowingsection,wewillgointomorespecificdetailontheresultsforthedescribedindicatorsandsub-scales. Table10:TableofDivergentCreativeSkillsDifferences,HighSchoolStudents,Hypothesis1a
MeasurementResultsforHighSchoolStudents
Averages,PretoPost Notes
Changeinnumberofdistinctproblemsidentified
Thetreatmentgroupshowedahighermeannumberofviableproblems(approximatelytwiceasmanyonaverage)thanthecontrolgroup,butthatdifferencedisappearedoncepre-sciencewascontrolledfor.
ControlPre:1.7ControlPost:2.3TreatmentPre:1.5TreatmentPost:2.9
Variabilityinbothgroupswashigh.Thiswasduetoanumberofparticipantswholostgroundfromthepretesttotheposttest.Twiceasmanycontrolgrouphighschoolstudentslostground,asdidtreatmentgroupstudents.
Changeinnumberofideaclusterstheproblemsrepresent.
Treatmentgrouphadagreaterchangeinthenumberofideaclusters.
ControlPre:3.2ControlPost:2.9TreatmentPre:3.0TreatmentPost:3.4
Changeinnumberofdistinctsolutions
Highschoolstudentsintreatmentgroupchangedmoreinnumberofdistinctsolutionslisted.
ControlPre:3.6ControlPost:3.4TreatmentPre:2.7TreatmentPost:3.7
Changeinnumberofideaclustersthesolutionsrepresent
Nosignificantdifferenceinchangebetweengroups.
ControlPre:2.7ControlPost:2.7TreatmentPre:2.3TreatmentPost:2.7
Thetreatmentgroupmademoregains.Nostatisticallysignificantdifferenceislikelybecausethevariabilityisconsiderable,andlargerinthecontrolgroup.
Changeinnumberofclustersofreasonsfortheselectionofthesolution
Treatmentgroupgainedsignificantlyinthenumberofclustersofreasonsofferedcomparedtothecontrolgroup.
ControlPre:1.4ControlPost:0.7TreatmentPre:0.7TreatmentPost:0.9
Itmaybethatthesignificantdifferenceisduetocontrolslosinggroundonthismeasureratherthantreatmentgroupmembersgainingsignificantground.Thisisthefirstmeasureinwhichsomanytreatmenthighschoolstudentsshowedstronggains;wouldbegoodtoinvestigatethismeasuretounderstandwhytheymighthavedoneso.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 43
Table11:TableofConvergentCreativeSkillsDifferences,HighSchoolStudents,Hypothesis1b
MeasurementResultsforHighSchoolStudents
Averages,PretoPost Notes
Changeinclarityofproblemstatement
Treatmentgroupsgainedmoreincreatingbetterproblemstatements.
ControlPre:1.3ControlPost:1.6TreatmentPre:0.9TreatmentPost:1.5
Theconfidenceintervalofthedifferencebetweenmeansshowedthatthedifferencebetweenthegroupswasasmuchasanentirepoint,whichisquitelargeona3-pointscale.
Changeinstrengthofsolutionstatement
Treatmentgrouphadmoregainsinprovidingstrongersolutionstatementscomparedtothecontrolgroup.
ControlPre:1.7ControlPost:1.8TreatmentPre:1.2TreatmentPost:1.8
Theconfidenceintervalofthedifferencebetweenmeansshowedthatthedifferencebetweenthegroupswasaboutoneentirepoint.
Changeinnumberofreasonsfortheselectionoftheproblem
Highschoolstudentsinthetreatmentgroupgainedsignificantlymoreinnumberofrationalesoffered.
ControlPre:0.7ControlPost:0.2TreatmentPre:0.2TreatmentPost:0.4
Generallyspeaking,highschoolstudentsinthecontrolgroupdidnotincreasetheirnumberofrationalesgivenfrompretopost.
Changeinnumberofreasonsfortheselectionofthesolution.
Treatmentgroupgainedmorethanthecontrolgroupinthenumberofrationalesfortheirsolutions.
ControlPre:1.4ControlPost:0.7TreatmentPre:0.7TreatmentPost:0.8
Whiletherewasaslightincreaseintreatmentpretesttoposttest,thesignificanceresultedfromthedecreaseinthecontrolgroup.
Changeinspecificityofwhatthesolutionis
Nosignificantdifferenceinchangebetweengroups.
ControlPre:0.9ControlPost:1.0TreatmentPre:0.8TreatmentPost:1.0
Neitheragainnoralossonthismeasure.
Changeinspecificityofhowthesolutionwillbeenacted
Treatmentgroupchangedmoreinspecificityinhowtoenactasolution.
ControlPre:0.2ControlPost:0.1TreatmentPre:Lessthan0.1TreatmentPost:0.1
Notethat81%showedneitheragainnorlossonthismeasure.Onaverage,thecontrolgrouplostground,thetreatmentgroupgainedground.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 44
Table12:TableofDivergentCreativeSkillsDifferences,EarlyCareerSTEMProfessionals,Hypothesis1a
Measurement
ResultsforEarlyCareerSTEMProfessionals
Averages,PretoPost Notes
Changeinnumberofdistinctproblemsidentified
Nosignificantdifferenceinchangebetweengroups.
ControlPre:2.3ControlPost:2.6TreatmentPre:1.4TreatmentPost:2.5
Whiletherewasnostatisticallysignificantdifferenceinthemeans,theadulttreatmentgroupsawalargergainonthismeasurepretopost,whilemoreofthecontrolgrouplostgroundfrompretopost.
Changeinnumberofideaclusterstheproblemsrepresent.
Nosignificantdifferenceinchangebetweengroups.
ControlPre:3.8ControlPost:3.7TreatmentPre:3.6TreatmentPost:3.5
Changeinnumberofdistinctsolutions
Nosignificantdifferenceinchangebetweengroups.
ControlPre:3.6ControlPost:4.2TreatmentPre:3.4TreatmentPost:3.4
Changeinnumberofideaclustersthesolutionsrepresent
Nosignificantdifferenceinchangebetweengroups.
ControlPre:2.6ControlPost:3.2TreatmentPre:2.4TreatmentPost:2.6
Thecontrolgroupmadefractionallymoregainsfromthepretesttotheposttest,onaverage,thandidthetreatmentgroup.Theamountofvariabilitywasconsiderable.
Changeinnumberofclustersofreasonsfortheselectionofthesolution
Nosignificantdifferenceinchangebetweengroups.
ControlPre:1.2ControlPost:1.4TreatmentPre:1.2TreatmentPost:1.1
AudienceViewpointsConsultingArtofScienceLearningResearchReport 45
Table13:TableofConvergentCreativeSkillsDifferences,EarlyCareerSTEMProfessionals,Hypothesis1b
Measurement
ResultsforEarlyCareerSTEMProfessionals
Averages,PretoPost Notes
Changeinclarityofproblemstatement
Trendwherethetreatmentgroupgainedmoreincreatingbetterproblemstatements.
ControlPre:1.4ControlPost:1.5TreatmentPre:1.0TreatmentPost:1.5
Thisoutcomemayhavehitstatisticalsignificance:1)iftheeffecthadbeenalittlestronger,2)thereweremanymoreparticipants,or3)therewaslessvariabilityinthedata.
Changeinstrengthofsolutionstatement
Nosignificantdifferenceinchangebetweengroups.
ControlPre:1.7ControlPost:1.7TreatmentPre:1.6TreatmentPost:1.7
Thedifferencewentinthedirectionofslightlyhighercreativityamongthetreatmentadults,comparedtocontroladults,whoaveragednogainonthisdifferencemeasure.However,thedifferencewasnotsignificant.
Changeinnumberofreasonsfortheselectionoftheproblem
Nosignificantdifferenceinchangebetweengroups.
ControlPre:0.7ControlPost:0.8TreatmentPre:0.6TreatmentPost:1.0
Thismeasurewasdivisiveforcontroladults:somegained,otherslost.
Changeinnumberofreasonsfortheselectionofthesolution.
Nosignificantdifferenceinchangebetweengroups.
ControlPre:1.2ControlPost:1.4TreatmentPre:1.2TreatmentPost:1.2
Changeinspecificityofwhatthesolutionis
Nosignificantdifferenceinchangebetweengroups.
ControlPre:1.1ControlPost:1.0TreatmentPre:1.0TreatmentPost:1.0
Highdiversityinscoresamongtheadultsbutthecontrolgroupshowedmorelosses,overtime.
Changeinspecificityofhowthesolutionwillbeenacted
Asmallnon-significantdifferencebetweenthecontrolandtreatmentadultsonthismeasure.
ControlPre:0.4ControlPost:0.3TreatmentPre:0.3TreatmentPost:0.1
Onaverage,bothcontrolandtreatmentgroupslostasmallbitofground.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 46
CriticalThinkingWeusedBasadurandFinkbeiner’s(1985)scaleonpreferenceforideationandtendencyforprematurecriticalevaluationofideas.ThisscalewasusedinRuncoandBasadur’s(1993)studyonideationandevaluativethinkingchangeduringworkplace-basedinnovationtraining,anddemonstratedadifferencebetweencontrolandtreatmentgroups.Eachoftheseitemswithinthescalewasscoredinbothpreandpostsurveys,andthentheresultswereusedtogenerateameanscoreforbothpreandpost.Theamountofchangeinmeanscoreforthetreatmentgroupswasthencomparedagainsttheamountofchangeforthecontrolgroups.Forthehighschoolgroups,thetreatmentgroupmadesignificantlygreaterpre/postgainsthanthecontrolgroup(seeTable14).Theimpactwaslarge;withthetreatmentgroupscoringasmuchas.75pointshigherthanthecontrolgrouponsomeitems.Thecorrelationmatrixshowedthatforthesecohorts,pre-artandpre-sciencescorescorrelatedstrongly,directlyandsignificantlywitheachother,r(63)=.52,p<.01.However,thecriticalthinkingdifferencedatadidnotcorrelatesignificantlywithpre-art,r(59)=-.14,p=.29,orwithpre-science,r(59)=.10,p=.43.Therefore,highschoolstudentdifferenceswereexaminedwithanindependentsamplest-test.Therewasasignificantdifferenceinthedirectionofgreatergainsmadebythetreatmentgroup,with32participants,comparedtothecontrolgroup,with29participants,t(59)=-2.84,p<.01.Theeffectofthearts-basedlearningwaslarge,Glass’Delta=.70.The95%confidenceinterval(CI)forthemeandifferences,-0.69,-0.11,showedthatthecontrolgroupscoredasmuchasthree-quartersofapointloweroncriticalthinkingthanthetreatmentgroup,respectively.FortheearlycareerSTEMprofessionals,therewasnosignificantdifferenceinchangebetweenthetwogroups(seeTable14).Thecorrelationmatrixshowedthatnoneofthecorrelationsweresignificant(pre-artandpre-science:r(67)=.11,p=.39;pre-scienceandcriticalthinkingdifference:r(67)=-.02,p=.88;pre-artandcriticalthinkingdifference:r(67)=.13,p=.29.Therefore,earlycareerSTEMprofessionalsgroupdifferenceswereexaminedwithanindependentsamplest-test.Therewasnosignificantdifferencebetweenthetreatmentandcontrolgroup,t(67)=1.35,p=.18.Thedifferencebetweenthetreatment(with37participants)andcontrolgroup(with32participants)wasasmuchashalfapointbutdidnotreachsignificance,95%CI;-0.10,0.51.Amongthetreatmentgroup,someadultsgainedasmuchasawholepoint(maximum=1.07)whereassomeotheradultslostnearlyawholepoint(minimum=1.00).Insum,thismeansthatboththearts-basedandthetraditionalinnovationtraininghaddivisiveeffectsoncriticalthinkingforadults,inthatsomebenefitedgreatlybutotherslostground.Table14:CriticalThinkingScore(BasedonBasadurandFinkbeiner)HighSchoolStudent
ResultsHighSchoolStudent
NotesEarlyCareerSTEMProfessionalResults
EarlyCareerSTEMProfessionalNotes
Significantdifferenceinthedirectionofgreatergainsmadebythetreatmentgroupcomparedtothecontrol.
Theeffectwaslarge,showingthatthecontrolgroupscoredasmuchasthree-quartersofapointlowerthanthetreatmentgroup.
Nosignificantdifferenceinchangebetweengroups.
Amongthetreatmentgroup,someadultsgainedasmuchasawholepointwhereassomeotheradultslostnearlyawholepoint.Bothtreatmentandcontrolhaddivisiveeffectsoncriticalthinkingforadults,inthatsomebenefitedgreatlybutotherslostground.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 47
CreativeThinkingSkillsSelf-Report
CreativeCompetenciesInventory(mini-ECCI) Oneofthemeasureswithinourpre-postassessmentsforCreativeThinkingwasthemini-ECCI.TheECCIstandsforEpsteinCreativityCompetenciesInventoryforIndividuals,andwasdevelopedspecificallyasaninstrumentformeasurable,trainablecompetencies.Previousstudies(Epstein,Schmidt,&Warfel2008)havesuggestedthatcreativeoutputcanbeincreasedthroughworkonstrengtheningthefollowingfourcompetencies:1)capturing(preservingnewideasastheyoccur),2)challenging(takingondifficulttasks),3)broadening(seekingknowledgeandskillsoutsideone'scurrentareasofexpertise),and4)surrounding(seekingoutnewstimuliorcombinationsofstimuli.TheversionoftheECCIusedherewasamini-version,designedforusewithindividuals,andwithfeweritemsthantheoriginal.Forthehighschoolstudents,thetreatmentgroupshowedsignificantlygreatergains,whenstatisticallycontrollingfortheeffectofpre-scienceexposure(seeTable15).Interestingly,themoreCreativeCompetencygainsparticipantsshowedfrompretopost,thelesstheyreportedpre-artorpre-scienceexposure.Pre-sciencebutnotpre-artwasasignificantcovariate.Thismightimplythathighschoolstudentswithlessexperienceinartorsciencepriortotheworkshoparemorelikelytoshowshort-termmeasurableimpactsfromarts-basedinnovationtraining.FortheearlycareerSTEMprofessionals,therewerenosignificantdifferencesinchangebetweenthecontrolandthetreatmentgroups.Neitherpre-artnorpre-scienceexposureshowedsignificantcorrelationsfortheadults.Overall,creativity,asmeasuredbytheECCIscale,significantlyincreasedinthehighschooltreatmentgroup,anddecreased(thoughnotsignificantly)withinthecontrolgroup(seeTable16).Table15:ECCIHighSchoolStudentsPre/PostChangeScoreBetweenGroups
MeanControl SD MeanTreatment SD Sig?
ECCIChangeScore -0.6 4.8 4.1 9.2Yes
(p<.05) Table16:ECCIHighSchoolStudentsPre/PostChangeScore
MeanPre SD MeanPost SD Sig?
Control 59.3 7.1 58.7 6.2 No
Treatment 60.6 7.5 64.7 9.1Yes
(p<.05)Thedifferenceinchangescoreswasnotduetothegroupsbeginningatadifferentstartingpoint,astherewasnostatisticaldifferenceintheprescoreforthetwogroups(seeTable17).
AudienceViewpointsConsultingArtofScienceLearningResearchReport 48
Table17:ECCIHighSchoolStudentsPreScoreComparisonBetweenGroups MeanControl SD MeanTreatment SD Sig?
ECCIPreScore 59.3 6.9 60.6 7.5
No PreandpostscoresfortheearlycareerSTEMprofessionalshadfairlysimilarmeans(seeTable18).Inboththecontrolandthetreatmentgroups,ECCIscoreswentdowninthepostassessment,significantlyso.TheECCIscorefellmoreinthecontrolgroupthaninthetreatmentgroup,andthedifferencebetweenthepreandthepostscoreswassignificant.Incomparingtheamountsofthedeclinebetweencontrolandtreatment(seeTable19),therewerenosignificantdifferencesinthechangescores.Whilethecontrolgrouphadbegunhigherthanthetreatmentgroup,thiswasnotstatisticallysignificant(seeTable20).Table18:ECCIEarlySTEMProfessionalsPre/PostChangeScoreWithinGroups
MeanPre SD MeanPost SD Sig?
Control 63.2 6.2 60.9 6.2Yes
(p<.05)
Treatment 61.7 6.1 59.9 5.4Yes
(p<.05)Table19:ECCIEarlySTEMProfessionalsPre/PostChangeScoreComparisonBetweenGroups
MeanControl SD MeanTreatment SD Sig?
ECCIChangeScore -2.3 6.1 -1.8 4.2 No Table20:ECCIEarlySTEMProfessionalsPreScoreComparisonBetweenGroups
MeanControl SD MeanTreatment SD Sig?
ECCIPreScore 63.2 6.2 61.7 6.1 No Overall,therewasnotanincreaseinthecreativityaspectsasmeasuredbytheECCIscaleintheearlycareerSTEMprofessionals.Infact,therewasastatisticallysignificantdecreaseincreativityinbothtreatmentandcontrolgroups.
CreativeProblemSolvingProfile(CPSP)Anotherscalewithinthepre-postassessmentwastheCreativeProblemSolvingProfile(CPSP)developedbyBasadur,Graen,andWakabayashi(1990).Thescalemeasuresindividualstrengthwithinfourdifferentcomponentsofthecreativityprocess:generation,conceptualization,optimization,andimplementation.Eachphasehasuniqueattributes.Ageneratorcreatesoptionsintheformofnewpossibilitiesornewproblemsthatmightbesolvedandnewopportunitiesthatmightbecapitalizedon.Aconceptualizercreatesoptionsintheformofalternatewaystounderstandanddefineaproblemor
AudienceViewpointsConsultingArtofScienceLearningResearchReport 49
opportunity,andgoodideasthathelpsolveit.Anoptimizercreatesoptionsintheformofwaystogetanideatoworkinpracticeanduncoveringallofthefactorsthatgointoasuccessfulimplementationplan.AnImplementercreatesoptionsintheformofactionsthatgetresultsandgainacceptanceforimplementingachangeoranewidea.Oneindividualmayhaveamixofthesestrengths.Basaduretal.(1990)assertthatdifferentindividualshavestrengthswithindifferentphasesofthecreativeprocess,andthatthisscalemeasurestheirrelativestrengths.Thisknowledgecanhelpthemimprovewithinareasofthecreativeprocessorchoosetocontributetheireffortstocertainphasesofcreativitythataremoreproductive.Theyholdthatcreativitytrainingcanimproveindividuals’strengthswithinthesedomains.Whenwecomparechangescoreswithinthehighschoolstudents(seeTable21),therewerenosignificantdifferencesbetweenthetreatmentandcontrolgroups.Table21:CPSPHighSchoolStudentsPre/PostChangeScoreComparisonBetweenGroups
Sub-score MeanControl SD MeanTreatment SD Sig?GenerationChangeScore
-0.8 5.7 -1.2 6.5 No
ConceptualizationChangeScore
1.3 3.1 1.2 4.5 No
OptimizationChangeScore
-0.2 3.5 -1.3 3.8 No
ImplementationChangeScore 0.7 4.7 1.4 4.2 No
Withingroups,thecontrolgroupdidexperienceastatisticallysignificantincreasewithintheconceptualizationsub-score(seeTable22).Table22:CPSPHighSchoolStudentsPre/PostChangeScore,ControlGroupOnly
Sub-score MeanPre SD MeanPost SD Sig?GenerationChangeScore
24.8 4.9 24.7 4.8 No
ConceptualizationChangeScore 19.3 2.9 20.6 3.6 Yes
(p<.05)OptimizationChangeScore 23.6 2.5 23.4 3.9 No
ImplementationChangeScore 26.1 3.5 26.8 3.8 No
AudienceViewpointsConsultingArtofScienceLearningResearchReport 50
Thetreatmentgrouphadnosignificantlydifferentscoresonthesub-scales(seeTable23).Table23:CPSPHighSchoolStudentsPre/PostChangeScore,TreatmentGroupOnly
Sub-score MeanPre SD MeanPost SD Sig?GenerationChangeScore 27.2 3.6 26.0 5.0 No
ConceptualizationChangeScore 21.3 3.1 22.5 3.7 No
OptimizationChangeScore 25.0 3.4 23.7 3.7 No
ImplementationChangeScore
26.6 3.7 28.0 4.3 No
Onthissamesub-scorewheretherewasanincreaseinthecontrol,therewasalsoadifferenceinwherethegroupsbegan(seeTable24).Thetreatmentgroupbegantheprogramsignificantlyhigherthanthecontrolgroupinconceptualization.Table24:CPSPHighSchoolStudentsPreScoresComparisonBetweenGroups
Sub-score MeanControl SD MeanTreatment SD Sig?GenerationChangeScore 25.0 5.0 27.2 3.6 No
ConceptualizationChangeScore 19.3 2.9 21.2 3.2
Yes(p<.05)
OptimizationChangeScore
23.9 2.5 25.0 3.3 No
ImplementationChangeScore
26.1 3.5 26.4 3.7 No
Thelackofstatisticallysignificantdifferencesamongthehighschoolstudentcohortswhenmakingadirectcomparisonbetweencontrolandtreatmentscoresforthefoursub-scoresontheCPSPmeansthatnoneofthefoursub-scalesshowedadistinctadvantagepre/postforonecohort(controlortreatment)overtheother.Whenlookingjustatthedifferencescoreswithineachgroup,thetreatmentgroupdidnotshowanysignificantdifferencesfrompretopostforthefoursubscales.Thecontrolgroupshowedonesignificantdifferenceinthesubscales,asignificantincreasefortheconceptualizationchangescore.Onefactorinthisdifferencewasthatthecontrolgroup’smeanpretestscorewassignificantlylowerafullpointthanthetreatmentgroup,whichleftmoreroomforthecontrolgrouptoimprove.TheCreativeProblemSolvingProfilecouldbeseenasmeasuringasetofskillsthatcouldbeverydifficulttochangeinashortperiodoftimegiventhattheyrepresentaperson’sgeneralapproachtothecreativityprocess.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 51
TherewerenosignificantdifferencesbetweenchangescoresofthecontrolandtreatmentgroupsforearlySTEMprofessionals(seeTable25).Table25:CPSPEarlySTEMProfessionalsChangeScoreComparisonBetweenGroups
Sub-score MeanControl SD MeanTreatment SD Sig?GenerationChangeScore -1.8 2.8 -1.7 3.1 No
ConceptualizationChangeScore -0.2 2.8 -0.6 3.1 No
OptimizationChangeScore -0.6 3.5 -1.2 2.6 No
ImplementationChangeScore
-0.7 2.9 -1.1 3.7 No
ThecontrolgroupdidhaveaslightdeclineintheGenerationsub-score(seeTable26),asdidthetreatmentgroupofearlycareerSTEMprofessionals(seeTable27).EarlycareerSTEMprofessionalsalsoexperiencedaslightdecreaseinontheoptimizationsub-scale.Table26:CPSPEarlySTEMProfessionalsPre/PostChangeScore,ControlGroupOnlySub-score MeanPre SD MeanPost SD Sig?GenerationChangeScore 27.9 3.7 26.1 3.7
Yes(p<.01)
ConceptualizationChangeScore 20.9 2.6 20.7 2.9 No
OptimizationChangeScore
24.0 3.3 23.4 2.2 No
ImplementationChangeScore
26.4 3.9 25.8 3.7 No
Table27:CPSPEarlySTEMProfessionalsPre/PostChangeScore,TreatmentGroupOnly
Sub-score MeanPre SD MeanPost SD Sig?GenerationChangeScore 28.1 3.6 26.4 3.8
Yes(p<.01)
ConceptualizationChangeScore 20.9 2.0 20.3 3.0 No
OptimizationChangeScore
24.3 3.2 23.2 2.4 Yes(p<.05)
ImplementationChangeScore
27.0 3.1 25.9 3.9 No
AudienceViewpointsConsultingArtofScienceLearningResearchReport 52
TherewerenostatisticallysignificantdifferencesinthestartingpointsofeachofthesegroupsontheCPSPscales(seeTable28).Table28:CPSPEarlySTEMProfessionalsPreScoresComparisonBetweenGroupsSub-score MeanControl SD MeanTreatment SD Sig?GenerationChangeScore
27.9 3.7 28.3 3.7 No
ConceptualizationChangeScore 21.0 2.5 20.9 2.0 No
OptimizationChangeScore 24.2 3.3 24.3 3.2 No
ImplementationChangeScore 26.6 4.0 27.0 3.1 No
TransferabilityofSkillsIncreasingindividualcreativethinkingskillsandhelpinglearnersdevelopattributestopreparethemforthe21stCenturySTEMworkplacearecommongoalsinmanyinformalSTEMlearningprojects.Nonetheless,theArtofScienceLearningprojectstandsoutintheproject’sfocusoninnovationandtotheextenttowhichitaimstochangepracticeinthehome,workplace,andelsewhere.Asshowninthecommentselsewherewithinthisreport,someindividualstookhomethelessonsandexperiencesfromtheirinnovationtrainingandreportedapplyingthemtoawiderangeofissues.TheAVCteamwishedtomeasurehowwellthoseskillsweresynthesizedandappliedelsewhereinparticipants’lives.Fourmonthsaftertheparticipantshadcompletedtheirinnovationtraining,theywereaskedtofilloutapost-workshopsurveyandreflection,askingtheextenttowhichthelessonstheylearnedduringthetrainingcouldbeappliedtoothercontexts,bothcurrentandfuture.Thequestionswerevariationsof“TowhatextenthaveyoubeenabletoapplytheInnovationChallengeexperiencetoyourcurrentworkorvolunteeractivities?”Thesewerepost-testitemsonly.Responseswereratedonascaleof1to7,with7indicatingthehighestleveloftransference.AcompositetransferabilityofskillsscorewascalculatedasanaverageofthefiveitemslistedinTable29.Forthehighschoolstudents,thetreatmentgroupratedthetransferabilityoflessonsfromthechallengetocurrentandfuturecontextssignificantlyhigherthandidthecontrolgroup.Theeffectwasverylarge;theactualdifferencebetweenthetwogroups’assessmentsrangedasmuchastwopoints.Thissuggeststhatthosewithinthetreatmentgroupexperiencedamuchgreatertransferenceoftheskillsintotheireverydaylives.Oneitem,usinga7-pointscale,askedtheextenttowhichtheparticipantswouldapplytheirexperiencestofutureworkandvolunteering:1outof5(20%)inthecontrolgroupratedita1“notatall,”whilenotoneperson(0%)inthetreatmentgroupratedita1,2or3.Conversely,while1outof5(21%)inthecontrolgroupsaidtheywereverylikely(a6or7)toapplytheirexperiencestofutureworkandvolunteering,peopleinthetreatmentgroupwerethreetimesmorelikely(65%)tosaytheywereverylikelytodoso.Anotheritemusingthesamescaleaskedtheparticipantsaboutapplyingtheirexperiencestoschoolorextracurricularexperiences:again,1outof5(20%)inthecontrolgroupratedita1“notatall,”andagainnooneinthetreatmentgroupratedita1,2or3.Ontheotherendofthescale,onlyoneoutofthree(30%)inthecontrolgroupratedita6or7,whiletwiceasmany(60%)inthecontrolgroupthoughttheywouldbeverylikelytoapplytheseexperiencestoschoolorextracurricularactivities.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 53
Itisimportanttonote,however,thereweremanymissingvaluesfromtheindividualswhochosenottofilloutthefinalsurvey.Asapproximatelyhalfofeachofthecontrolandtreatmentgroupsdidnotfilloutthesurvey,itispossiblethisfindingisanartifactofthesampleresponding.Furtherworkwoulddowelltoresearchthisaspectoftransferabilitymorethoroughly.FortheearlycareerSTEMprofessionalsgroupstherewerenostatisticallysignificantdifferencesinthecompositescorebetweenthecontrolandtreatmentgroups(seeTable29).Withintheparticipantsthereweremanymissingvalues,uptohalftheoriginalsamplewhocompletedthefive-weekworkshop.Thisquestionshouldbereplicatedwithinanyfurtherworkinthisvein.Table29:TransferabilityofSkillsLearningAverageScores
HighSchoolStudent
AveragesEarlyCareerSTEM
ProfessionalAveragesTowhatextenthaveyoubeenabletoapplytheexperiencetoyourcurrentworkorvolunteeractivities?
Control:3.7Treatment:4.8
Control:4.1Treatment:3.6
Towhatextentdoyouthinkyouwillbeabletoapplytheexperiencetoyourfutureworkorvolunteeractivities?
Control:4.1Treatment:6.0
Control:4.5Treatment:4.0
Towhatextenthaveyoubeenabletoapplytheexperiencetoyourcurrentschoolorextracurricularactivities?
Control:4.2Treatment:5.2
Control:3.3Treatment:3.2
Towhatextentdoyouthinkyouwillbeabletoapplytheexperiencetoyourfutureschoolorextracurricularactivities?
Control:4.1Treatment:5.8
Control:4.2Treatment:3.6
Towhatextenthaveyoubeenabletoapplytheexperiencetoyourcurrenthome/personallife?
Control:3.0Treatment:3.7
Control:3.1Treatment:3.2
AudienceViewpointsConsultingArtofScienceLearningResearchReport 54
DefinitionofInnovation
HighSchoolStudentsHighschoolstudentparticipantswereaskedtowritetheirdefinitionofinnovationaspartofthePreWorkshopSurvey.Individualshadawidevarietyofresponsesthatfellintoafewdifferentcategories(seeTable30).Mostoften,theseparticipantsdescribedinnovationas“newideas”(55%);thenextmostcommondefinitionofinnovationinvolvedreferencestoproblemsandsolutions(25%).Table30:DefinitionofInnovationPreSurvey,HighSchoolStudents
Control(n=33)
Treatment(n=32)
Total(n=65)
Newideas 58% 53% 55%Referencetosolutionsand/orproblems 24% 25% 25%Uniquenessofidea 15% 19% 17%Processofinnovation–brainstorm,reframing,iterating,experiment,collaboration,strategicthinking,etc.
9% 19% 14%
Beingcreative 12% 12% 12%Workethic/workapproach,personalcharacteristicsoftheinnovator 3% 12% 8%
Productimplementation,bringingtomarket/world,adopter,incremental 0% 3% 1%
Miscellaneous 12% 6% 9%Themostpopulardefinitionofinnovationgivenbyhighschoolstudentswasasnewideas.Morethanhalfofthehighschoolstudents(55%)includedthisthemeaspartoftheirdefinition. MydefinitionofInnovationistobringinnewideas.
Innovation,inmyopinion,istheuseofnewideasandsolutionsforagreater,goodpurpose.Mydefinitionofinnovationiscreatingsomethingnewtohelpothers.
Onequarterofstudents(25%)definedinnovationintermsofsolutionsandproblems.
Mydefinitionofinnovationwouldbetheprocessofbrainstormingandputtingtoworkideasandtheories.
Mydefinitionofinnovationishavinganew,better,andstrongersolutiontoaproblemthatisputinfrontofyou.Formeinnovationisexperimentingwithaspecificprobleminordertosolveit.Iseeitasawayofmakingthingsbetterforourworld,howeveritdoesn'tneedtoonlybemechanical,itcanbeemotionalaswell.
Somehighschoolstudents(17%)wroteabouttheuniquenessofanideaasadefinitionofinnovation.
Mydefinitionofinnovationisanewideathatnobodyhaseverthoughtof.Anideathatstands
AudienceViewpointsConsultingArtofScienceLearningResearchReport 55
outfromtherest.However,standingoutisnotthemostimportantelement.Themostimportantelementisputtinganeffortintoone'swork. Innovationistakinganideaandmakingitwilder.Takingtheoldandthenewandformingsomethingoutofthisworld.Innovationisbeingunique,beingdifferent,notfollowingthenormbutratherstrengtheningtheweirdandwacky,itstakingtheidea'softheinsaneandfindingasolutiontoaproblemwiththem,inventinganewmachinewiththemandbetteringhumanitywiththem.
Somestudents(14%)usedwordsabouttheprocessofinnovationtodefinetheirideas.
Innovationisdevelopmentofaconceptorideainordertoimproveasituationorwayoflife.
Someonewhointroducesandcommunicatesanovelwayofthinking,interpreting,producing,orsolving.
Ithinkthatinnovationisusingnewideasthatareusuallyunrelatedtoacertainsituationtosolveaproblemthathasarisen.Ialsothinkitissomewhatofatrialanderrorprocess,workingthingsoutuntiltheyareperfect.
Afewstudents(12%)thoughtthatbeingcreativewasadefinitionofinnovation.
Innovationisyourabilitytobeoriginal,createnewthings,likeanideaorproject.Itstronglycorrelateswithcreativity.Mydefinitioniswhenoneusestheirresourcestocontractandsuggestnewandcreativeideasormethods.Innovationiswhenyoucancreativelyorartisticallyallowyourselftochallengeyourselftocreatenewideas.
Afewstudents(8%)wroteabouttheworkethicandpersonalcharacterofaninnovator. Mydefinitionofinnovationissomeonewhoworkshardandisproductiveandorganized. Tomethatworkinghardandbringingcleverideaswillleadtoaninnovator.Averysmallnumberofstudents(1%)definedinnovationinveryhigh-levelterms,includingproductimplementationandbringingproducttomarket.
Creatingnewwaysandcomingupwithnewideasandtakingthemtoawholenewlevel,meaningtoputthemintogooduseandgoodservicetothecommunity.
Andafewhighschoolstudentsgaveresponsesthatdidnotfitintoanyofthesecategories.
Isthatinnovationisthatyoustartanideafromzeroandleadittobecomehero.Thenextupandcoming.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 56
EarlyCareerSTEMProfessionalsEarlycareerSTEMprofessionalparticipantswerealsoaskedtowritetheirdefinitionofinnovationaspartofthePreWorkshopSurvey.Individualshadawidevarietyofresponsesthatfellintoafewdifferentcategories(seeTable31).Theseresponsesshowedabroadsimilaritytothoseofthehighschoolstudentcohorts.Mostoften,fortheearlycareerSTEMprofessionals,participantsdescribedinnovationas“newideas”(58%);thenextmostcommondefinitionofinnovationinvolvedreferencestoproblemsandsolutions(39%).Table31:DefinitionofInnovationPreSurvey,EarlyCareerSTEMProfessionals
Control(n=32)
Treatment(n=37)
Total(n=69)
Newideas 59% 57% 58%Referencetosolutionsand/orproblems 53% 27% 39%Processofinnovation–brainstorm,reframing,iterating,experiment,collaboration,strategicthinking,etc.
28% 22% 25%
Uniquenessofidea 19% 27% 23%Beingcreative 16% 11% 13%Productimplementation,bringingtomarket/world,adopter,incremental
12% 5% 9%
Workethic/workapproach,personalcharacteristicsoftheinnovator 3% 0% 1%
Miscellaneous 3% 8% 6%Themostpopulardefinitionofinnovationgivenbyadultparticipantswasasnewideas.Morethanhalfoftheadultsincludedthisthemeaspartoftheirdefinition. Idefineinnovationasanythingnew. Toinventorbegintoapplynewmethodsorideas
Anewidea,anewwayofdoingsomething.Overonethirdofadults(39%)definedinnovationintermsofsolutionsandproblems.
Innovationislookingataproblem,generatingdifferentsolutions,andfindinganoutcomethatimprovesthecurrentstate.Developingnewapproachesormethodstoreachsolutions.Innovationistakingsomethingyoualreadyhaveorknowandapplyinganewmethodtosolveaproblem.
Onequarterofadults(25%)usedwordsabouttheprocessofinnovationtodefinetheirideas.
Innovationistheprocessofbrainstorminganddevelopingneworuniquesolutionstoexistingissues.Theinnovationdoesnotnecessarilyhavetobesubstantialinnature.Evenincrementalinnovationcanleadtomassiveimprovements.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 57
Innovationisideationthatcomestofruition--itisthewholeprocessfromdefiningtheproblemtobrainstorming,toevaluating,toselecting,toexecutingasuccessfulsolution.Itisusedtosolveimmediateproblemsthatinhibitwork/school/progress.Innovationiscombiningcreativityandknowledgetodevelopideas/products/solutionsthathavethepotentialtoaffectmanypeoplein(hopefully)apositiveway.Innovationhasnolimits.
Almostonequarterofadultparticipants(23%)wroteabouttheuniquenessofanideaasadefinitionofinnovation.
Thinkingoutsidetheboxtocomeupwithnovelsolutionsortotakefromoutsidedisciplinesorideasinordertobettersolveanissueorproblem.Youdon'tneedtorecreatethewheel,younearlyneartomakeitbetter.Usingcreativemethodsforthinkingandcollaborationtogeneratenewandusefulideasandsolutionstodailyproblems.Oftenaprocessthatleadstobetterefficiencyintheworkplaceandwithinpersonalspacesaswell.Developingorcreatingsomethingnovel.
Someadults(13%)thoughtthatbeingcreativewasadefinitionofinnovation.
Usingthecreativeprocesstodevelopnewideasorprocesses.
Innovationisoriginalthoughtasitrelatestocreativewaystosolvedifficultproblems.Theseproblemsareonesthatimpactcommunities,socreativesolutions/innovationsarenecessarywhentherearesomanystakeholdersinvolved.Thinkingofnewwaysforthingstowork.Creativeproblemsolving.
Afewadultparticipants(9%)definedinnovationinveryhigh-levelterms,includingproductimplementationandbringingproducttomarket.
Whenyouturnanideaintoaproductormethodthatisbetterthanothersthatalreadyexists.
Anewmethod,idea,orproduct,putontrialwiththehopeofsuccessandacceptanceofscienceorartintheeyeofthepublic.
Averysmallnumberofadults(1%)wroteabouttheworkethicandpersonalcharacterofaninnovator.
Innovationcantakemanyforms.Peoplewhoareinnovativearenotafraidtochallengethenorm.Theytakeapartanissue,wraptheirheadsaroundit,andthengooutintospaceandlookattheprobleminitsentirety.Inanutshell,Ibelievethegenerationofanynewwaytotackleaproblemcanbeclassifiedasinnovation.Innovationcanalsobethemodificationofexistingknowledge.Comingfromanengineeringbackground,Icanseeanyincreaseinefficiencyorproductionasinnovative.
AndafewearlySTEMprofessionalsgaveresponsesthatdidnotfitintoanyofthesecategories.
Orderoutofchaos...
AudienceViewpointsConsultingArtofScienceLearningResearchReport 58
SelfPerceptionasInnovatorThisstudyhadmanyscalesoncreativityandcriticalthinking,whichtheprojectteamhypothesizedwouldincreaseinindividualsoncetheyhadhadtrainingininnovation.Asthetrainingsubjectmatterwithinboththetreatmentandthecontrolgroupsfocusedoninnovation,weaskedparticipants,preandpost,whethertheyperceivedthemselvestobeinnovatorsatwork,school,andhome.Thequestionwasaskedona1to7scale,with1representingStronglyDisagreeand7representingStronglyAgree.Withinthehighschoolstudentcohorts,therewerenosignificantdifferencesbetweencontrolandtreatmentgroups,buttherewasaslighttrend(seeTable32).Thistrendwasnotduetoanincreaseinthetreatmentgroup.Rather,thecontrolgroupmeanshowedamildloss,treatmentgroupdidnotshowagainorloss.FortheearlycareerSTEMprofessionals,therewasnotmuchdifferencebetweenthepretestandposttestscoresforeitherthetreatmentorcontrolgroupininnovatorself-assessmentsamongadults,butwhatdifferencestherewere,tendedtobelessintheposttest. Table32:ChangeinSelfPerceptionofInnovation(InnovatorSelf-AssessmentDifferenceScore)
HighSchoolStudentResults
HighSchoolStudentAverages
EarlyCareerSTEMProfessionalResults
EarlyCareerSTEMProfessionalAverages
Trend.Therewasanon-significantdifferenceinscoresbetweengroups.
ControlPre:5.8ControlPost:5.1TreatmentPre:5.9TreatmentPost:5.9
Trend.Therewasanon-significantdifferenceinscoresbetweengroups.
ControlPre:5.4ControlPost:5.3TreatmentPre:5.4TreatmentPost:5.3
HighSchoolStudentsDiggingfurtherintotheperceptionofinnovation,weaskedparticipantsinboththepreandthepostsurveywhytheyhadgiventhemselvestheratingstheydid(seeTable33).Inadditiontotalkingaboutnewanduniqueideas(41%),highschoolstudentsalsocitedtheirabilitytobeinnovatorsintheirpersonallife(28%)andintheirschoolworkandprofessionalwork(21%).Table33:SelfPerceptionofInnovationPreSurvey,HighSchoolStudents
Control(n=33)
Treatment(n=32)
Total(n=65)
Newideas/uniqueideas 39% 44% 41%Iaminnovativeinmypersonallife/hobbies 33% 22% 28%Mywork/schoolinvolves/requiresinnovation 27% 16% 21%Ichooseadifferentpath,creativesolution,creativeoutput
9% 31% 20%
Referencetosolutionsand/orproblem 3% 16% 9%Mywork/schooldoesnotallowmetobeaninnovator 9% 6% 8%
Idon’thavetimeathometobeinnovative/Idon’twanttochangethingsinmypersonallife 0% 0% 0%
Miscellaneous 21% 12% 17%
AudienceViewpointsConsultingArtofScienceLearningResearchReport 59
Highschoolstudentsmostoften(41%)referredtonewideaswhentalkingaboutthemselvesasaninnovator.
Iseemyselfasaninnovatorinmanyregards.I’malwayslookingforcoolnewideasandIalwaysfindthemintheunlikeliestofplaces.I’malwayslookingtoimprovesomething,I’malwaysbusywithonethingoranother.ItendtostartprojectsbeforeIfinishothers.Iliketodothingsthathaveneverbeendonebefore.Also,Iliketo“gobigorgohome”.
Yes,IseemyselfasagoodsampleofaninnovatorasItrytobringnewthingseveryday.
IliketointroducenewideasandnewstrategiestoeverythingIworkwith.Whetherit’sschoolorworkorpersonal,Ilovebringinginnewconcepts.
Morethanaquarterofstudents(28%)felttheywereinnovativeintheirpersonallifeandhobbies.
WhenitcomestothingslikeschoolworkIliketodoitinunconventionalways.UnlikemostIdon’tjustsitdownanddomyhomework.Ineedtobedoingsomethingtogetmyselftobetterformideas.WhenitcomestomyhobbieshoweverIamalwaystryingnewthingsandexperimentingwithdifferentways.Forexamplewhenitcomestomybeat-boxingI’malwaystryingtocomeupwithnewtechniquesandsoundstoaddtomyarsenaltomakemeabetterbeat-boxer.
Isometimesthinkofeasier,simplerwaysordoingeverydaythings.ButIrarelycomeupwiththingstodowithschool.
IstronglyagreethatIamaninnovatorinregardstomypersonallifebecauseIhavehadtocomeupwithwaystokeepwhatIholddearinmybusyschedule.Icookalmosteveryday,andIhavebegunrecreatingtraditionallyunhealthyrecipestohealthieralternatives.Myfavoritesofarisacookiecakemadewithgarbanzobeansandmaplesyrupinsteadoftraditionalwhiteflourandsugar.Oneofmycriteriaisthatithastotastegreat,anditalwaysdoes!
Overonefifthofthehighschoolstudents(21%)reportedbeinginnovativeintheirschoolworkandprofessionalwork.
IseemyselfasagoodexampleofaninnovatorbecauseI’malwaysopenandenthusiasticaboutnewideas.Duringmysummerjob,IbecameefficientinthemethodstheytaughtmetoputcasefilesintothecomputerandthenIcreatedmyownmethodthatwasmoreefficientthantheirs.Iseemyselfdoingthisinmylifeaswell.IlovetakinganideaandimprovingonitandIseemyselfdoingthisinmyeverydaylife.IfI’mnotrememberingthematerialtaughtinschool,IfindamoreeffectivewayformetoprocesstheinformationwhetherthisbechangingthewayIstudyorthewayIlearn(avisualvs.verballearner).
Ifindmyselfaveryinnovativeandresourcefulperson.Atwork,(lifeguardingandrestaurantwaitstaff)Itendtolookforsolutionstoproblemsandthinkingofnewwaystodocertaintasksuntilthebestsolutionhasbeenfound.Ialsotendtoincorporatesimilarideasintomypersonallifeandespeciallymyhobbies(tennis,piano).Inschool,Itrytobeasinnovativeaspossible,butIdefinitelyfeelrestrictedduetocurriculumrequirements.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 60
InmyschoolworkIoftenfindcompletelydifferentwaysofdoingassignments,waysthattheteacherneverthoughtofwhencreatingtheproject.
Somestudents(20%)wroteaboutbeinginnovativebychoosingadifferentpathoracreativesolutionintheirlife.
OftentimesIgoaboutadifferentapproachtoaproblemversusadifferentapproachmypeershavechosen.
IbelieveIamaninnovatorbecauseIusuallytakeadifferentapproachthanothersattemptingtocreatemoreefficientwaysofproducingasolution.IseemyselfasaninnovatorbecauseIfocusoncreativityinmylife.Ifollowthisbecausetheworldwouldnotbeuniquewithoutdifferences.
Afewstudentparticipants(9%)talkedaboutinnovationintheirlifebydefiningproblemsandfindingsolutions.
IfeellikeIamaninnovatorbecauseI’malwaystryingtofindnewsolutionsfortheproblemsthatIface,especiallyinmypersonallifeandatschool.SinceIdon’thaveaprofessionallife,myschoolandmyhomearetheonlyplaceswhereIcantrytochange,butI’malwayslookingfornewplaceswhereIcantrytochange.
AslongasIhaveareallygoodfeelforwhattheproblemI’maddressingis,andhavesomesenseofthehistoryoftryingtosolvetheproblem(previousattemptstosolve),IliketothinkI’minnovativeenoughtohelpprogresstowardsasolution,ifnotsolveaproblem.Inmyshopatschool,Biotech,wefacealotofproblemsandareconstantlytryingtosolvethem,likemakingbacteriafluoresceunderUVlightsandsomanyotherexperiments.
Itendto,nomatterthesituation,trytodevelopaneasiersolutionthatcanbenefitmeandothersaroundme.
Others(8%)felttheycouldnotbeinnovativeintheirschoolorprofessionalwork.
Idon’tfeelthatschoolworkgivesmetheopportunitytoinnovateasmuchasidlike.Ihaveabusinesswithmymomandthatgivesmealotofartisticinnovation.AndIamanathleteandmusicianandIcomeupwithnewplaysandcomposemusic,whichletsmeinnovatealot.Myschoolworkisnotvery“innovatable”tobeginwith,myworkisdeskjob,andmyinterestsareopenenoughtocreateapersonalstyletogoaboutthem.
17%oftheparticipantsgaveresponsesthatdidnotfitintoanyofthesecategories. IseemyselfasaninnovatorinmostchallengesIamfacedwith.
Myratingsonwhetheriamainnovatoristhatiamorganizedandtrytounderstandeverything
AudienceViewpointsConsultingArtofScienceLearningResearchReport 61
IratedmyselfthatsinceIbelieveIamnotthebestnortheworstinnovatorandthatitdependsonthesituationatthetime.Ikindofwantedtodolaserworkinthefuture,likebuildingthatkindofstuffinthefuture.Idon’tseemyselfasaninnovator.Iprefertohearotherpeople’sideasandagreeordisagreewiththem.SometimeswiththoseideasIwouldaddmoreontothem.
HighSchoolparticipantswereaskedtorateandexplaintheirself-perceptionasaninnovatoragainaftercompletingtheInnovationChallenge(seeTable34).Table34:SelfPerceptionofInnovationPostSurvey,HighSchoolStudents
Control(n=33)
Treatment(n=32)
Total(n=65)
Iaminnovativeinmypersonallife/hobbies 39% 47% 43%Ichooseadifferentpath,creativesolution,creativeoutput
24% 31% 28%
Newideas/uniqueideas 33% 16% 25%Mywork/schoolinvolves/requiresinnovation 18% 31% 25%Referencetosolutionsand/orproblem 12% 28% 20%Mywork/schooldoesnotallowmetobeaninnovator 18% 16% 17%
Idon’thavetimeathometobeinnovative/Idon’twanttochangethingsinmypersonallife 0% 3% 1%
Miscellaneous 18% 12% 15%Thehighschoolstudentsmostoften(43%)saidtheyfelttheywereinnovatorsintheirpersonallifeandhobbies.
IgavetheratingsIdidbecauseIusedifferenttechniquestofindasolutioninschool.And,inmypersonallife,Ienjoydrawing,whichiscreativeandinnovative.
IdonotfeelIammuchofaninnovatorwhenitcomestoschoolworkbecausetheassignmentsarewithinrestrictions.TherearesomeprojectswherecreativitydoesallotandIcanbeinnovative.Ontheotherhand,IconsidermyselfasaninnovatorinmypersonallifebecausethosesettingsareopenendedandIinnovatepassionatelyforwhatIamdoing.AtschoolIalwaysworkinawaythatdefiesthenormandreallysurprisesmyteachers.InmypersonallifeIamconstantlytryingoutnewthingsinallmyhobbies,aswellasnewhobbies,usingtrialanderrortodevelopmyownwayofdoingthings.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 62
Morethanaquarterofstudents(28%)identifiedasinnovatorsbecausetheychoseamorecreativesolutionorpath.
I'maverycreativeandimaginativeperson;Itendtowonderinmymindalot(whenthetimeisrightofcourse)yetIfindthatthepeoplearoundmejustknowmeasacreativeperson.IconsidermyselfaninnovatorbecauseI'malwayslookingfornewthingsandtryingtochangewhatisaroundme.Iliketoseethingsinadifferentperspectiveandwonderwhatwouldhappenifthingsweredifferent.
Onequarterofhighschoolstudents(25%)describedthemselvesasinnovatorsbecauseoftheirneworuniqueideas.
IconsidermyselftobeaninnovatorbecauseI’mconstantlythinkingaboutalternateideasandwaysthatcanhelpineverydaylife.IconsidermyselfasaninnovatorbecauseIalwaysliketotakeonnewchallengesandplanorgetanoriginalideaoutofit.Iliketothinkofnewideasandmethodstodothings,andnotjustfollowdirections
Onequarterofhighschoolstudents(25%)believedthemselvestobeinnovatorsintheirschoolwork,orfeltthattheirschoolworkrequiredinnovation.
Inschoolwork,Iamaninnovatorbecausemyprocrastinationisabigproblemforme,soIestimatehowlongaprojectthatIhaveforaweekwouldactuallytakeme.Ifinreality,theprojecttakesmetwodays,Iletmyselftoprocrastinateuntiltherearetwomoredaysleft.InmyschoolworkIalmostalwaysgoaboveandbeyondtohavesomethingnew,better,greater,tohavethebestpossible"thing"possible,youseeeverythingyoudoisareflectionofyousodoitthebesteverydaybecauseputtingsomethingoffonlyhurtsyou.IwilltakeasimpleDNAmodelneededforbiologyandmakea4foottallDNAmodelchemicalandregularthatmovetogether;howeverthepartsarecommonlyusedthings.InmyhobbiesaswellasinanyareaofmylifeIdon'tfollowtherulespersayIdoeverythingoutofthebox,takesoccerforexampleIdon'tjustplayconventionallyIalwaysseenewwaysofdoingsomething.Icreatenewwaysforthingsthatareseeminglyaone-solutionproblem.InmyschoolworkandpersonallifeIhavedefinitelyseenashiftinthewayIgoaboutdoingsomething.I'malwayslookingfordifferentwaystogettoasolutionandevencomingupwithnewerwaystodothings.Someofmycoachesandteachershaveactuallyincorporatedmyideasintothewaytheyteachsomethingtomyteamandinclass.
Onefifthofstudents(20%)madereferencetofindingsolutionstoproblemsintheirlives.
IconsidermyselftobeaninnovatoratmyschoolworkandinmypersonallifebecauseItendtobeaproblemsolvertonoextent,IalwaystrynewsolutionsandnewideasbutalsoIenjoyusingoldsolutionsthatwork,stickingtothebasics.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 63
Itakedifferentapproachestosolutionsthanpeoplenormallydo.AndIamconfidentinmywaytocommunicate.
Ispendalotoftimelookingforthequickest,mosteffectivewaytocompletebothinandoutofschooltasks.Icreatethingsorputideastogethertosolveproblems.
Somehighschoolstudents(17%)felttheirschoolorworkdidnotallowthemtobeinnovative.
InschoolIdon'thavetoomanyopportunitiestobeinnovative.InmymindIcanbecreativeandexpressthings.
Itendtobelessinnovativeinschoolbecauseteacherswillsometimessetlimitsthatwemustconformtoinordertosucceed,whichcanbeverystifling.Inmypersonallife,idon'tlimitmyselftoanythingandIammoreconfidentbeingmyownpersonwhenihavecontroloverhowicanbesuccessful.
Averysmallnumberofstudents(1%)chosenottobeinnovativeintheirpersonallife,orfelttheydidn’thavetheopportunitytobeinnovativeintheirpersonallife.
Inschoolwork,Icomeupwithnewsolutionsandnewwaystodotheworkonmyown,IfindmyownmethodsandIlearnbestthisway.Inmyhobbies,suchasbasketball,I'mworkingonestablishedplays,notdevelopingmyownplays.Ivaluehowthecoachtellsmetodoit.Butinschool,Ifindthereismoreflexibilitytobeinnovativeanddomyownthing.
Andafewhighschoolstudentsgaveresponsesthatdidnotfitintoanyofthesecategories. FeelmorefreetosayanythingIthought. IgavethisratingbecauseIfeellikeIamainnovator Incomparingthechangesinthehighschoolstudents’perceptionsofthemselvesasinnovatorsbetweenthepreandposttests,therewerebroadsimilaritiesbetweencontrolandtreatmentgroups.Onenotabledifferencewasinthenumberofparticipantswhomentionedwaysinwhichtheywereinnovativeintheirownpersonalandprofessionallives.Thepercentageoftreatmentparticipantswiththisresponsemorethandoubled(22%pre/47%post)whilethecontrolgroupshowedonlyaveryslightincreaseinthiscategory(33%pre/39%post).Interestingly,thecategorysayingthatworkorschooldoesnotallowthemtobeaninnovatordoubledfrompretopostforthecontrolparticipants(9%pre/18%post)andmorethandoubledforthetreatmentparticipants(6%pre/16%post).
AudienceViewpointsConsultingArtofScienceLearningResearchReport 64
EarlyCareerSTEMProfessionalsBeforethestartoftheInnovationChallenges,eachparticipantwasaskedtoratehimorherselfasaninnovator,andthenaskedtoexplaintheratinginhisorherownwords(seeTable35).Table35:SelfPerceptionofInnovationPreSurvey,EarlyCareerSTEMProfessionals
Control(n=32)
Treatment(n=37)
Total(n=69)
Mywork/schoolinvolves/requiresinnovation 62% 49% 55%Iaminnovativeinmypersonallife/hobbies 28% 38% 33%Ichooseadifferentpath,creativesolution,creativeoutput 41% 19% 29%
Newideas/uniqueideas 22% 35% 29%Mywork/schooldoesnotallowmetobeaninnovator
0% 30% 16%
Referencetosolutionsand/orproblem 9% 19% 14%Idon’thavetimeathometobeinnovative/Idon’twanttochangethingsinmypersonallife
16% 6% 10%
Miscellaneous 6% 5% 6%MorethanhalfoftheearlycareerSTEMprofessionals(55%)reportedbeinginnovativeintheirschoolworkandprofessionalwork.
Inmyprofession,Iamresponsibleforcreatingandimplementingmanyenjoyablelessonplansandengagingcurriculum.Irarelyreuselesson,thereforehavetocreatenewactivitiesregularly.
Iseemyselfastakingontasksrelatedtoourresearchanddeterminingthebestwaystosolvetheissue.Forexample,weworkedtodevelopacalibrationschemetoproduceaccurateandreliabledataforatmosphericwatermeasurements.
Itrytodonewthingsandchallengemyselfwithdifferentanduniqueprojectsatschool,home,andwork.
Onethirdofadults(33%)felttheywereinnovativeintheirpersonallifeandhobbies.
Idon'tfeelthatI'mcomingupwithnewideasfortheclassroom,justputtingmyownspinonthings.IdothinkthatIcanbemorecreativeinmypersonallifethough.Iliketodoandtryallsortsofnewthings!
Ifeelmuchmorepassionateaboutmyhobbiesandpersonallifeversusmyprofessionallifeand,assuch,ammorewillingtotrynovelthingstokeepthingsinterestingandprogressive.I'moftentoldthatwhenI'minterestedinsomething,Ireallygetintoit.ThisofteninvolvescreatingsomethingoutoftheordinarywithregardtovarioushobbiesIhave.
Someadultparticipants(29%)wroteaboutbeinginnovativebychoosingadifferentpathoracreativesolutionintheirlife.
Student:Ihaveacreativeedgewhenitcomestoschoolengineeringprojects.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 65
InmyprofessionallifeIhavehadtopulltogethermaterialsandgetcreativewithlessonplanning.InmypersonallifeIhavehadtobecreativegatheringequipmentfordifferenthobbies.Iliketoresearchideas-whetherexperiments,Pinterest,art,recipes-thenspinofffromthemtocreatesomethingnew.IalsocreatenewthingswhenIresearchtheinternetandcan'tfindwhatIamlookingfor.
Someadults(29%)referredtonewideaswhentalkingaboutthemselvesasaninnovator. Alwayslookingfornewapproachesonoldthings.
IliketothinkIamaninnovatorbecauseIliketocomeupwithnewideasonhowtodothisandchallengemyselftoit.Ithinktestingoutthingsthatmightnotalwaysworkoutisgoodandcanleadtoprogress.
AfewearlycareerSTEMprofessionals(16%)felttheycouldnotbeinnovativeintheirschoolorprofessionalwork.
Atwork,Imaythrowoutideasoften,butIamnotinapositiontoputmyideasintomotion-itisuptosomeoneelse.IamalsonotoftenputinasituationwhereIneedtocomeupwithsolutionstoproblemsatwork.Inpersonallife,IlovetocomeupwithnewideasforthehobbiesIaminterestedin.Idon'thavemuchroomforflexibilityinmyjobbutIdocomeupwithwaystomakemyjobrolemoreefficientandeffective.Idon'tseemyselfasagreatinnovatorinmyschoolworkbecausetheclassesItakealldon'trequiretoomanydifferentwaystoapproachthesubject.ItsverystraightforwardandIdon'ttrytofindawaytothinkofitdifferentlyoreventrytorelatetoanothersomeothersubjects.
Afewadultparticipants(14%)talkedaboutinnovationintheirlifebydefiningproblemsandfindingsolutions.
Ienjoybecomingcompetentinmyfieldsofinterest.IknowI'matahighlevelwhenIcanprovideinnovativeacceptablesolutionstodifficultsituations.Iseemyselfasaninnovatorwhenbrainstormingideastosolveuserexperienceissuesrelatedtotechnology.Ienjoyengagingindesignthinkingandbouncingideasoffotherpeople.However,Iamalsoafanofthetriedandtruesolution.Ifsomethingexiststhathaspreviouslybeendonebeforeandworkseffectivelythenitshouldbeutilizedratherthanforcinganewinnovation.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 66
Afewadults(10%)felttheydidnothavetimetobeinnovatorsintheirpersonallife,orchosenottobeinnovativeintheirpersonallife.
Iseemyselfascreativewithmyjobandfindingnewwaystoteach.Inmypersonallife,IfeellikeIhavemyroutinesandsometimesgetstuckinthem.
Inbothschoolworkandmyprofessionallife(academic)thereisapremiumplaceduponinnovation,definedasnewideastosolveproblems.Inmypersonalinterests,andmypersonallife,Itendtoavoidtheadoptionofaproblem-solvingattitude.
Andafewadults(6%)gaveresponsesthatdidnotfitintoanyofthesecategories. Youcaneithertalkaboutit...orbeaboutit! Iliketobeproactiveandaforewordthinker.AftercompletingtheInnovationChallengeparticipantswereagainaskedtorateandexplaintheirself-perceptionasaninnovator,toseeifindividualsreportedanychangeintheirresponse(seeTable36).Table36:SelfPerceptionofInnovationPostSurvey,EarlyCareerSTEMProfessionals
Control(n=33)
Treatment(n=32)
Total(n=65)
Mywork/schoolinvolves/requiresinnovation 50% 38% 43%Newideas/uniqueideas 37% 27% 32%Iaminnovativeinmypersonallife/hobbies 34% 27% 30%Mywork/schooldoesnotallowmetobeaninnovator 22% 22% 22%
Idon’thavetimeathometobeinnovative/Idon’twanttochangethingsinmypersonallife 22% 16% 19%
Ichooseadifferentpath,creativesolution,creativeoutput
25% 8% 16%
Referencetosolutionsand/orproblem 12% 13% 13%Miscellaneous 9% 13% 12%EarlycareerSTEMprofessionalsmostoften(43%)believedthemselvestobeinnovatorsintheirschoolwork,orfeltthattheirschoolworkrequiredinnovation.
Idoresearchinauniversitylab.NearlyeverydayIgettocomeupandtestoutnewideas-thingsthathaveneverbeendonebefore.Atwork,I'madesigner.AlldayIhavetocreativelyandefficientlysolveanythingfromsmallproblems,tolargeinvolvedproblems.Constantlykeepinginmindwhatourcompetitionisdoingandresearchingnewtechnologiesandapplications.Athome,Iamalwaystinkeringwithmyjewelrybusinessorotheraround-the-housetasks
Myworkrequiresmetobeinnovativeeverydaytosolveproblemsthathavenosolutions.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 67
Almostonethirdofadultparticipants(32%)describedthemselvesasinnovatorsbecauseoftheirneworuniqueideas.
Ialwaystrytothinkoutsideofthebox.Iamconstantlythinkingofnewinventionsofnewtechnologyandjottingthemdowninmyphone.
IconsidermyselfsomewhatofaninnovatorbecausewhenIhaveideasIliketestingthemoutanddon'tshootthemdownrightaway.Itryoutnewactivitiesandideasofferedtomeinmylife,butIalsofeelthat,asastudent,Idonothavemuchopportunitytobeamajorinnovatorinworkorpersonallife.
AlmostonethirdofearlycareerSTEMprofessionals(30%)saidtheywereinnovativeintheirpersonallifeandhobbies.
Idon'thavemuchcreativefreedomatwork.Withmyhobbies,Iamabletodevotemoretimetocomingupwithnewideas.Ifeelthatmyhobbiesallowformorecreativeexpression(danceandaerialacrobaticperformance)andinnovationindevelopingnewmovements.Atwork,therearemorerulestofollow.
Almostonequarterofadultparticipants(22%)felttheirschoolorworkdidnotallowthemtobeinnovative.
Atwork,itisdifficulttobeaninnovatorwhensomanyrulestofollowarealreadyinplace:forexample,underaprofessorduringresearchorasanEMT,youdon'treallygettostrikeoutonyourown.Inmypersonallife,however,Iwouldconsidermyselfmuchmoreofaninnovatorthanaverage,asIlovetoexplorenewareasofscience,literature,andoutdooradventures,astheseexperienceshelppreventmefromgettingbored.Idon’tdoanythingmeaningfulatwork.Myworklacksinterestandrecognitionforinnovation.Inmypersonallife,Iknowmycapabilitiesforexperience.
Someadults(19%)chosenottobeinnovativeintheirpersonallife,orfelttheydidn’thavetheopportunitytobeinnovativeintheirpersonallife.
Ialwaystrytoimprovemylessonsplansandimplementnewstrategieswithmystudentseachyear.Inmypersonallife,IknowwhatIlikeandtendtostickwiththosethings.IratedmyselffairlyhighontheinnovatorscaleinmyworklifebecauseIalwaystrytothinkofwaystoimproveprocessesatwork.Inmypersonallifethough,Idon'tthinkItrytocomeupwithnewideasthatoften.Itrynewexperiences,butIdon'tthinkofwaystoimprovethoseexperiences.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 68
ProfessionallyIhavesupervisorsthatImustanswertosoalotofmy"innovationatwork"needstofallwithinparametersthatmeettheirgoals/expectationsofaprogram.Withthatbeingsaid,Iworkinasafespacewherenewideasareacceptedandsupported.IthinkIhavemoremainstream/basichobbies/interestsoutsideofworksoIdon'tthinkI'mallthatinnovativeinmypersonallife.
Someadultparticipants(16%)identifiedasinnovatorsbecausetheychoseamorecreativesolutionorpath.
Irarelyseelimits.IamalwaysbeingtoldbyfriendsandfamilythatIimagineideasthatothersthinkarebrilliant--butneverwouldhavethoughtof.
IconsidermyselfaninnovatorinmyresearchbecauseIusedcreativethinkingtodevelopanewmethodinmyfieldofscience.Inmypersonallife/hobbies,Iamsomewhatofaninnovator.Itakecreative/newishapproachestosomeofmyartsyandathletichobbies.Forexample,Icreatedmyownexercisehobbythatisamixofbellydancing,tai-chiandbalanceboarding.
Afewadults(13%)madereferencetofindingsolutionstoproblemsintheirlives.
IseemyselftobeaninnovatorbecauseI'vebeenabletostepbackonmyparticipationinanorganizationandfigureoutwaysforpublicitytobedramaticallychangedandhelped.MyideasbuildoffsomeotherexistingonesbutithasatwistonitanddefinitelycaterstomydemographicandI'mabletoseetheproblemgettingtheminvested.
IfeelIamaninnovatoratworkbecauseIdonotliketoworkharder,butsmartersoItrytofindnewwaystoworkataproblembutdifferently.
Iworkasanengineerandoftenhavetocomeupwithanewideaforanygivenproblem.IworkwithmanyproblemssoIhavemanyopportunitiestocomeupwithnewsolutions.AthomeIloveworkingonartprojectsandrestorationprojectsthatrequireinnovativesolutionstoproblems.
AndafewearlycareerSTEMprofessionalsgaveresponsesthatdidnotfitintoanyofthesecategories.
MybiggeststrugglewithsayingIamaninnovatorismystrugglewiththeconceptofnewness.Iliveinmyheadandidesleakoutofmybody!IconsideranalmostgraduatinginnovatorbecauseIhaveyettoproduceaproduct,butIamstilllearningto.
IncomparingthechangesintheadultcareerSTEMprofessionals’perceptionsofthemselvesasinnovatorsbetweenthepreandposttests,therewereevenmoresimilaritiesthanwiththehighschoolstudents,withnolargepercentageshiftsfrompretotheposttests.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 69
Asacomplementtothequestionofwhetherindividualsperceivedthemselvestobeinnovators,wealsoaskedparticipantsthefollowingquestion:“Hasyourperceptionofyourselfasaninnovatorchanged,evenalittlebit,duringyourparticipationinthisproject?”Forthoserespondingaffirmatively,weaskedtheopen-endquestion“inwhatways?”todeterminethenatureofthatperceivedchange.Therewerenosignificantdifferencesbetweenthehighschoolgroupstothefirstquestion.Bothgroupsrankedtheirperceptionofthemselvesasinnovatorsaschanged,withanaverageof85%inthecontrolgroupand81%inthetreatmentgroup.TheearlycareerSTEMprofessionalsalsohadnostatisticallysignificantdifferencesbetweengroups,though73%oftheadults(87%inthecontrolgroupand68%inthetreatmentgroup)respondedthatthecoursehadchangedtheirperceptionsofthemselvesasinnovators.SeeTable37belowtoseehowthosestudentswhoreportedachangeinself-perceptionasinnovatorscharacterizedthatchange,withsomenotablepatternsofdifferentiation.Table37:ChangeinPerceptionofSelfasanInnovator,HighSchoolStudents
Control(n=33)
Treatment(n=32)
Total(n=65)
Ihaveimprovedcertainbehaviors 61% 78% 69%Reframingtheprocessofinnovation 33% 34% 34%Collaboration 12% 28% 20%Divergentthought 12% 19% 15%Confidence 12% 9% 11%Findingproblem/solution 6% 9% 8%Leading/following 6% 3% 5%
Ihavethecapacityorinteresttobeaninnovator 9% 6% 8%InnovationisharderthanIthought,it’schallenging,Idon’tfeelasmuchlikeaninnovator
15% 0% 7%
Myperceptionhasnotchanged 6% 6% 6%Miscellaneous 3% 3% 3%Themajorityofhighschoolstudents(69%overall,61%control/78%treatment)reportedimprovementofcertainbehaviors,attitudesorskillsduetoparticipationintheInnovationChallenge.Breakingdownthoseareasofimprovementintomoredetail:Aboutonethirdofhighschoolstudents(34%)felttheirideasabouttheprocessofinnovationhadbeenreframedandimproved.
Ilearnedmoreaboutinnovationinthisproject,andIlearnedtheprocessusedtoproperlyfilterideasintoasolution.
Ithastaughtmetofocusonthecreativesidelonger.
IfeellikeitimprovedmywayofthinkingnowIapplyittoeverything.Onefifthofstudents(20%overall,12%control/28%treatment)saidtheircollaborationskillshadimprovedasaresultofparticipatingintheprogram.
BeforethisprojectIdidn'tthinkitwaspossiblethatIcouldworkinagroupandaccomplishaprojectbutnowIrealizeitwasjustthatI’vejustneverworkedonaprojectwithlike-minded
AudienceViewpointsConsultingArtofScienceLearningResearchReport 70
individualswhoactuallywanttobeworkingontheproject.
Ineverthoughtinnovationcouldbeacollaborativegroupeffort.Normally,Iampersonwhopreferstoworkalone.Thisgroupinnovationprocesshasreallyopenedmyeyestohowpeoplecancollaboratetogetsomethingdone.Thisskillisextremelybeneficialtomylifeespeciallymyacademiccareer.I'lldefinitelybringtheskillsIacquiredfromthisexperiencetocollegenextyear.Irealizedthatinnovationcanhappeninagroup,ratherthanjustinaperson.
Someparticipants(15%)believedtheirabilitytothinkdivergentlyhadimprovedaftertakingpartintheInnovationChallenge.
Icanthinkofmoreideasatonceandmoreangles.Ifeellikemyideasaremuchmorecoherentthantheywerebefore,Ithinkofallthepossibilities.Forexample,beforetheprojectIusedtowalkaroundhavingmyeyesfixedinjustonedirection,nowIlookatalldirectionsandItrytoseenewthingseverywhere.
Someparticipants(11%)felttheirconfidencehadimprovedasaresultofparticipatingintheexperience.
Ifeelmoreconfidentinmyabilitytoinnovate.Ifeelmuchmoreconfidentinmyabilitiesofmymindbecauseoftheideasme,alongwithmyteamcameupwith,alongwiththecapacityofmybraintoworkonmultipleproblemsatonce.IfeelIamaleaderandnotafollowerandwilltakeaction,whichIknewthese,buttheyarerenewedandgivenmenewfoundconfidence.IbelieveIhaveanewprocess.
Afewparticipants(8%)reportedanimprovementinfindingproblemsandsolutions. Icanthinkmoreclearlyaboutproblemsandopportunitiestofixthem. I'mmoreinclinedtothinkaboutproblemsinadifferentwaythanbefore.Afewparticipants(5%)reportedanimprovementinleadingand/orfollowing.
IhavelearntthatIcanletotherscontrolthesituationandthatIcansitbackandlistentoothersequallybrilliantideas.IhavelearnthowtostoppanickingaboutthelittlethingsandseethebiggerpictureandmostimportantlyIhavelearnthowtocommunicatewithothersinordertogetmypointacrossandintegratetheirideastothegrandschemeofthings.MyperceptionofmyselfhaschangedthroughoutthedurationofthisprojectbecausenowIunderstandhowtothinkofusingoldthingsorsystemsinnewexcitingways.AlsoIthinkaboutproblemsolvingandgroupworkdifferently.IhaveseenhowIhavechangedandbecomemoreofaleaderinmygroup.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 71
Outsideofthosehighschoolstudentswhosaidtheyhadimprovedcertainbehaviors:Afewparticipants(8%)felttheirself-perceptionasaninnovatorchangedbecausetheynowfeltliketheycouldbeaninnovator,comparedtotheirperceptionbeforeparticipatingintheInnovationChallenge.
Afterparticipatinginthisprojectandlearningthespecificsofwhatmakessomeoneaninnovator,Icanconfidentlylabelmyselfasaninnovator.
IactuallyfindthatIamevenmoreofaninnovatorthanIthought.Iwasn'tawarehowquickIamatthinkingupaplethoraofsolutions.
Afewparticipants(7%overall,15%control/0%treatment)learnedthatinnovationorbeinganinnovatorismoredifficultthantheyoriginallyperceived.
IhaverealizedthatIamlessofaninnovatorthatIthoughtIwas,oratleastthatmostofmyinnovationsareshort-termandrelativelyeasytoimplement.MyperceptionhaschangedbecausethisprojectwaschallengingattimesandIrealizedhowhardIactuallycanworkandhowmuchIactuallycaninnovatewhenIputmymindintoit.BeforeIdidthingsveryimpetuously,notreallythinkingthingsthrough.NowIknowtheprocessforinnovationandcansystematicallyworkthroughproblems.
Afewhighschoolstudents(6%)reportedthattheirperceptionofthemselvesasaninnovatordidnotchangeasaresultofparticipatingintheInnovationChallenge.
No,myperceptionofmyselfasaninnovatordidnotchangeduringmyparticipationinthisproject.Myperceptiondidn’treallychange
Andafewparticipantsgaveresponsesthatdidnotfitintoanyofthesecategories.
ItshowedmetoseeandrealizehowdisorganizedWorcesteris
IfeltasifIwaslettingthepanelreallyinterestedintomygroup'sinnovationsotheywouldwanttoseeitinthefuture.
EarlycareerSTEMcareerparticipantswerealsoaskedinwhatwaystheirperceptionofselfasaninnovatorhadchangedasaresultofparticipatingintheproject(seeTable38).ThemajorityofearlycareerSTEMprofessionalparticipantsmentionedanimprovementofcertainskillsafterparticipatingintheInnovationChallenge(54%).
AudienceViewpointsConsultingArtofScienceLearningResearchReport 72
Table38:ChangeinPerceptionofSelfasanInnovator,EarlyCareerSTEMProfessionals
Control(n=32)
Treatment(n=37)
Total(n=69)
Ihaveimprovedcertainbehaviors 59% 49% 54%Collaboration 19% 24% 22%Reframingtheprocessofinnovation 22% 19% 20%Confidence 16% 5% 10%Divergentthought 9% 3% 6%Leading/following 6% 3% 4%Findingproblem/solution 0% 0% 0%
InnovationisharderthanIthought/Idon’tfeelasmuchlikeaninnovator
6% 22% 14%
Myperceptionhasnotchanged 12% 16% 14%Ihavethecapacity/interesttobeaninnovator 9% 11% 10%Miscellaneous 6% 3% 4%ThemajorityofearlycareerSTEMprofessionals(54%overall,59%control/49%treatment)reportedimprovementofcertainbehaviors,attitudesorskillsduetoparticipationintheInnovationChallenge.Webreakdownthoseareasofimprovementintomoredetailbelow.OveronefifthofearlycareerSTEMprofessionals(22%)saidtheircollaborationskillshadimprovedasaresultofparticipatingintheprogram.
Thisprojectgavemeadeeperunderstandingofwhatitlookedliketopurposefullyinnovateasagroup.IthinkIhaveabetterunderstandingofthecollaborationprocessandquantitativestepsthatareinvolved.Inthepast,Iexperiencedgeneratingnewideasinagroupsetting.TheChallengemadeitvery/moreclearthatinnovationbyiteration(offotherpeople'sideas)generatesnewideasmuchfasterthanworkinginisolation.ThisissomethingI'veexperiencedbefore,butdidn'treallyrecognizethefullstrengthofworkingingroupsinthismanneruntiltheChallenge.
Onefifthofadultparticipants(20%)felttheirideasabouttheprocessofinnovationhadbeenreframedandimproved.
Ireallyappreciatedhowthoroughlywewentthroughtheinnovationprocess.Irecognizedtheflowofthehugeamountofideasgeneratedduringbrainstorming,tohoninginonasingleaspectofaproblem,andthendevelopingamultifacetedapproach,butIhadneverreallypinned/brokendownthoseseparatesteps.Ibuckedattheideaofastructuredformulaforinnovationatfirst,butthencametoacceptitasIcouldseeitworkinginthegroup.Ifeelwaymoreopentousingdifferentapproachestodevelopingideasandsolutions,especiallyusingmoreofanartisticapproachforwhatmayseemmorelikeascientificproblem-theuseofmusic,doodling,movement,etc.Ihavelearnedalotmoreabouttheprocessofinnovation,aswellasthewaythatIrespondtoit.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 73
Ihavelearnedhowtobeinnovativewithagroup.Someparticipants(10%overall,16%control,5%treatment)felttheirconfidencehadimprovedasaresultofparticipatingintheexperience.
I'mmoreconfidentinmythinking,honestly.IhadapositiveexperienceduringthewaterchallengeandwasproudthatsomeoftheideasIhadwereincorporatedintoourfinalproject.Enteringtheproject,Iwasn'tsureifI'dbeabletocontributemuchtotheinnovationprocesssinceIfiguredthatI'dbeoneoftheyoungestmembersintheproject.However,IwasexcitedtoseethatIcouldcontributeoriginal,innovativeideasbasedonmyownexperiencesinlife,soI'vegainedmoreconfidenceinmyselfasaninnovator.
Afewadultparticipants(6%)believedtheirabilitytothinkdivergentlyhadimprovedaftertakingpartintheInnovationChallenge.
Irealizethatyoudon’tneedtobethesmartestpersonintheroomortheloudesttobeinnovative.Lookingatdifferentideasevenifyouthinktheymaybecrazyatfirst(thehumanfecescompositingforexample)couldhavepotentialifthereissomeideastomakeitmainstreamenoughtohavelargergroupgainacceptance.It’sthebalanceofideasthathelpthewildideasgetclosertousablemarketableproducts.IsupposeIhaverealizedthatIcanbemorecreativeandcomeupwithmoreideasthanwhatIthoughtIcould.
Afewparticipants(4%)reportedanimprovementinleadingand/orfollowing.
Mostlylearningtoshareresponsibilitiesandarticulation.BeforeIdidn'tbelieveIwasinnovativebutintheprocessIfeltvalidatedbymyteambecausemyleadershipskillswereabletoshinebutalsomyactualengineeringskills.BecausemyteamchosetheideasthatIhelpeddevelopedIbelievethatIamsomewhatofaninnovator.Ialsofeellikewewereabletobuildoffeachother'sideastoo.
OutsideofthoseearlycareerSTEMprofessionalswhosaidtheyhadimprovedcertainskillsorattitudes:Someadultparticipants(14%)learnedthatinnovationorbeinganinnovatorismoredifficultthantheyoriginallyperceived. IsupposeIfeellessinnovative;Ihadmomentsofdesiringnoinnovationwhenfrustrated.
Beforetheproject,innovationwasabroadertermthatincludedanyeventthatoneusedtheircreativitytoimproveorsolveaproblem.Aftertheproject,innovationequalspressure.Thepressuretodevelopacompletelynewideathathasthepotentialtodowellinthemarketingworld.Thepressuretodevelopanideathatislifechangingtomorethanjustoneperson.Tome,innovationnowseemslikesomethingI'malmostincapableofachieving.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 74
ItmademefeellikeIwaslessofaninnovatorbecauseitseemedmyexistingperceptionsoftheproblemathandkeptallmyideasinaboxrelatedtohowpracticaltheywouldbeinreallife.
Someadults(14%)reportedthattheirperceptionofthemselvesasaninnovatordidnotchangeasaresultofparticipatingintheInnovationChallenge.
Ithasn't.Idon'tfeelIchangemywaytoseethingsduringthisproject.Iappeartohavelesspatienceforpoorlythoughtoutideas.
Afewadultparticipants(10%)felttheirself-perceptionasaninnovatorchangedbecausetheynowfeltliketheycouldbeaninnovator,comparedtotheirperceptionbeforeparticipatingintheInnovationChallenge.Andafewparticipantsgaveresponsesthatdidnotfitintoanyofthesecategories.
UsingChallengeSkillsintheFutureThelastquestionofthetransferabilityinstrumentfocusedonthefutureandparticipant’sabilitytoimplementnewskillsandknowledge(seeTable39andTable40).Participantswereasked,“WhatwastheonethingyouexperiencedintheChallengethatyouthinkwillbemosthelpfultoyouinthefuture,andwhy?”Theskillmostoftencitedbythehighschoolstudentcohortwascollaborationandteamwork(21%).TheskillmostoftencitedbytheearlycareerSTEMprofessionalscohortwasalsocollaborationandteamwork(27%). Table39:WhatWillbeMostHelpfulintheFuture,HighSchoolStudents
Control(n=33)
Treatment(n=32)
Total(n=65)
Collaborationskills 18% 28% 21%Process-basedinnovationskills 15% 25% 20%Divergentthinkingskills 0% 9% 5%Contentknowledge 6% 0% 3%Miscellaneous 3% 0% 1%Morethanonefifthofhighschoolstudents(21%)saidcollaborationskillswouldbethemostusefulthingtheylearnedduringtheInnovationChallenge.
Howtoproblemsolvewithgroupsbecauseitoccursineveryone'sdailylivesanditisahardtaskwithsomanydifferentopinionsandpointofviews.Teamworkskillsandnevershuttingdownsomeone'sidea,nomatterhowstrangeitis.Knowinghowtoworkbetterinagroupandeffectivelycommunicate.IthinkthiswillbemosthelpfulbecauseItendtoshyawayfromsharingmyopinion,butIlearnedbetter.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 75
Onefifthofhighschoolparticipants(20%)felttheprocess-basedinnovationskillswouldbethemostusefulthingtheywouldtakeawayfromtheChallenge.
Learningabouthowinnovationisaprocess,willhelpmefollowalongthatprocesswhenthetimecomesforprojectsandsuch.
Thefunnelingofpossiblesolutionstoanexactsolution. Brainstormingideasandideadevelopmentbecauseitimportanttotrynewthings.Afewstudents(5%)mentioneddivergentthinkingasaskilltheylearnedduringtheInnovationChallengethatwouldbehelpfultotheminthefuture.
Ithinkthinkingfromadifferentperspectivebecauseitopensupawholenewworldofideas.Afewstudents(1%)gainedcontentknowledgeduringtheInnovationChallengethatwouldbehelpfultotheminthefuture. Howtravelcanbeimproved.Andafewhighschoolstudentsgaveresponsesthatdidnotitintoanyofthesecategories. Thatwhenforcedtoworkwithidiots,nothinggetsdone.Table40:WhatWillbeMostHelpfulintheFuture,EarlyCareerSTEMProfessionals
Control(n=32)
Treatment(n=37)
Total(n=69)
Collaborationskills 28% 27% 27%Process-basedinnovationskills 28% 16% 22%Contentknowledge 9% 11% 10%Divergentthinkingskills 0% 0% 0%
Note:Visitorscouldprovidemorethanoneresponsetothisitemsothecolumnpercentagestotalmorethan100%MorethanonequarterofearlycareerSTEMprofessionals(27%)saidcollaborationskillswouldbethemostusefulthingtheylearnedduringtheInnovationChallenge(seeTable40).
UnifyingthoughtsandteammembersfromvaryingbackgroundsaroundatechnicalsolutionforaproblemthatdeeplyaffectsallindividualslivinginSouthernCalifornia.
Collaborationinaninterdisciplinaryteam.
Beingabletoworkwithagroupandgetthetaskdoneunderpressure.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 76
Morethanonefifthofadultparticipants(22%)felttheprocess-basedinnovationskillswouldbethemostusefulthingtheywouldtakeawayfromthechallenge.
Writingthebusinessplan-IhaveasmallsidebusinessthatI'mworkingongrowing,soI'llneedto. Theideatoiterate,design,iterate,anddesignagain.
Brainstormingactivities.Someadults(10%)gainedcontentknowledgeduringtheInnovationChallengethatwouldbehelpfultotheminthefuture.
Ilearnedthatwaterisapublicgood,notacommodity.IthinkthatwasveryessentialtoapproachthewaterproblemthatIdidn'tknowuntilWeek3.Thatledmetothinkaboutenergy,whichisacommodity,notapublicgood,andhowthatchangestheeconomicenvironmentbetweenthetwodifferententities.Outsideofthetechnicalbackground,thisenlightenedmetolookintootheraspectsthatcanbeverycriticaltomakingnewservices/productssuccessful.
Ihaveabetterunderstandingofwatershedsandhowtohelppreservethem. Knowledgeofhowthewatersystemworks.Unlikethehighschoolstudents,noearlycareerSTEMprofessionalsmentioneddivergentthinkingasaskilltheylearnedduringtheInnovationChallengethatwouldbehelpfultotheminthefuture.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 77
Hypothesis2Findings:CollaborationBehaviors Hypothesis2:Arts-basedinnovationtraining,comparedtotraditionalinnovationtraining,increasesindividualcollaborativebehaviorswithinateamcontext.Collaborationplaysacentralroleinvirtuallyallinnovationprocesses,whichisreflectedintheteam-basednatureoftheArtofScienceLearninginnovationcurriculum.Hypothesis2tookacloserlookattheimpactofarts-basedlearningonthecollaborativebehaviorsofinnovationteamsfacedwithincreasinglycomplexanddemandingcollaborativetasksoverafive-weekperiod.Collaborationwasresearchedbyobservingelevencollaborativebehaviorswithintheteams,aswellasbyaskingteamstoreflectonhowwelltheyworkedtogether,withoverlapbetweenitemsthatformedthebasisforobservationalandparticipantreports.Fortheobservedbehaviors,datacollectorsobservedteamswhentheywereworkingtogetherinanycapacity,suchasactivities,brainstorming,orontheirproblemsandsolutions.Asparticipantswerewearingnametags,datacollectorsspecificallyrecordedwhichoftheindividualstheysawengagingineachoftheelevenbehaviorsduringagivenobservationperiod,whichusuallywasbetween10and30minutes.Ifanyactivitylastedlongerthan30minutes,halfwaythroughobserverswouldswitchtoanotherteamsotobetterbalancetheamountofcollaborativeactivitythatwasobservedamongthevariousteams.Thetotalnumberofindividualswhoengagedinthebehaviorswasthentotaledforeachteampereachobservedblockofactivity;theseblockswerethenaddedtogethertoproducedataontotalnumberofobservedinstancesofeachactivityforeachoftheteamsthatday.Thislastnumberiswhatisrepresentedintheanalysesbelow.Fortheself-reportedteamratingmeasuresincludedinthissection,eachindividualfilledoutashortsurveyattheendofeachweekthatincludedratingsforthesamecollaborativebehaviorsbeingobserved.Fortheself-reportteamratingmeasures,individualsratedtheirteam(asopposedtoparticularindividualsontheirteam)oneachoftheelevenitems.Inthismanner,theobservationsandsurveyswereintendedtoprovidemultiplewaystomeasurethesamevariablesandacomparisoncouldbemadebetweenwhattheresearchersobservedandwhattheteammembersperceived.AppendicesCandDincludetheobservationalandself-reportedteamcollaborationratingmeasures,respectively.
ObservedCollaborationInordertodeterminewhetherornottherewasachangeinspecificcollaborativebehaviors,ananalysiswasruntodeterminewhethereachindividualshowedthatcollaborativebehaviormore,lessorthesameattheendoftheresearchperiodastheydidatthebeginning.Then,thepercentagebreakdownofthethreecategorieswascomparedforthetreatmentandcontrolgroupsindependently.Inotherwords,theanalysistoldwhethertherewasadifferenceforeithercohortfromthebeginningtotheendoftheirparticipation.Forthehighschoolstudentsthereweresignificantincreasesinpositivecollaborativebehaviorsinboththecontrol(6of11measures)andtreatment(5of11measures)conditions(seeTable41).WhenlookingattheearlycareerSTEMprofessionals,itwasmoreone-sided:thecontrolconditionshowedasignificantincreaseforonly1ofthe11positivecollaborativebehaviors,whilethetreatmentconditionshowedasignificantincreasefor7ofthe11.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 78
Table41:SummaryTableofObservedCollaborationandOtherInnovationProcessesDifferences(Pre/PostObservationalDifferenceScores) HighSchoolStudents EarlyCareerSTEMProfessionals
ObservationalCategory
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
Sharesleadership Yes Control Yes TreatmentActivefollower No - Yes TreatmentEmotionallyintelligentbehavior
Yes Treatment Yes Treatment
Empathiclistening Yes Treatment Yes TreatmentMutualrespect No - Yes Treatment,ControlTrustinmovingtowardsolution Yes Treatment,Control Yes Treatment
Transparentincommunication
Yes Control Yes Treatment
Abilitytodisagreeproductively Yes Treatment No -
Definingacommonpurpose
YesControl No -
Creatingcultureofmutualaccountability
Yes Control No -
Productivelymanagesdisruption
Yes Treatment,Control No -
Self-ReportedCollaborationFortheself-reportedteamcollaborationratingmeasures,individualsratedthemselvesonfifteencollaborativebehaviors;thiswasdoneduringapre-testinthefirstsession,andduringapost-testinthelastsession.Adifferencescorewascalculatedbysubtractingthepre-testratingorscoreonanitemfromthepost-testscoreonthatsameitem.Ifoneorbothoftheitemswasmissing,thenthatindividualwasnotincludedinthecomparison.Twostatisticallysignificantdifferencesemergedforthehighschoolstudents;thecontrolgroupwasmorelikelytoreportapositivechangefrompre-testtopost-testinmutualrespectandalsotrust(seeTable42).InlookingattheearlycareerSTEMprofessionalstherewerenostatisticallysignificantdifferencesinthepre-testpost-testchangescores.Thismeansthattheadultsdidnotperceiveanynotabledifferencesfromthebeginningofparticipationtotheendinthevariouscollaborationscores,eitherthoseinthecontrolortreatmentconditions.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 79
Table42:SummaryTableofSelf-ReportedTeamcollaborationRatingsandOtherInnovationProcessesDifferences(Pre/PostSelf-ReportedDifferenceScores) HighSchoolStudents EarlyCareerSTEMProfessionals
Self-ReportCategory
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
Sharesleadership No - No -Activefollower No - No -Emotionallyintelligentbehavior
No - No -
Empathiclistening No - No -Mutualrespect Yes Control No -Trustinmovingtowardsolution Yes Control No -
Transparentincommunication
No - No -
Abilitytodisagreeproductively
No - No -
Definingacommonpurpose
No - No -
Creatingacultureofmutualaccountability
No - No -
Productivelymanagesdisruption
No - No -
Successfullycompletedtask
No - No -
RelationshipBetweenObservedandSelf-ReportedCollaborationItisinterestingthattheobservedcollaborationmeasuresshowedamuchlargerimpactthantheself-reportedteamcollaborationratingmeasures.Althoughneithercontrolnortreatmentgroupsratedthemselvesashavingimprovedonanyoftheself-reportedteamratingmeasures,observersrecordedthetreatmentgroupsashavingengagedinmanyofthesebehaviorssignificantlymorebythelastsession.ThedifferencewasparticularlyevidentamongtheearlycareerSTEMprofessionals,butthehighschoolstudentsalsoshowedamuchhigherrateofsignificanceforobservedcollaborativebehaviorscomparedtotheirself-reportedteamratingbehaviors.Thesefindingssuggestthatsimplyaskingpeopleabouttheircollaborativebehaviorsmaynotbeenoughtoaccuratelyportraythenatureandextentofcollaborationoccurringduringgrouptasks;andthatthirdpartyobservationisimportanttounderstandingtheoccurrenceofthesebehaviors.
HighSchoolStudentObservedBehaviorFindingsThetablesinthefirstpartofthissection(Table43throughTable53)showtheobservedbehaviorratingsbytheresearchersfromR2(week2)throughR5(week5)andincludetotalbehaviorsperteampersession;seeAppendixCfortheindividualbehaviorsstudied.Atableisincludedforeachoftheelevenbehaviorsobservedbytheresearcher,breakingdownthenumberoftimesthisparticular
AudienceViewpointsConsultingArtofScienceLearningResearchReport 80
behaviorwasobservedforeachteamduringeachweek.Theoverallsumofbehaviorsobservedineachweekisincludedatthebottomofthetable.Note:R1(week1)observationaldatawerecollectedforthetreatmentgroup,butnotforthecontrolgroup.SincetheobservationaldataforhighschoolstudentsinR1areincomplete,R1hasbeenexcludedfromthetablesandfiguresbelow,aswellasallrelatedstatisticalanalyses.Inordertocomparethegroupsoneachbehavior,statisticaltestswererunonthedifferencesbetweenthefrequencyandpatternsofoccurrencethesebehaviorswereobservedincontrolandtreatmentovertheR2toR5period.Iftherewasastatisticallysignificantdifferencebetweenthecontrolandtreatmentgroups,alinegraphwasincludedbelowthetable.Tableswithoutaccompanyinglinegraphsshowednostatisticallysignificantdifferencesbetweencontrolandtreatment.BasedontheR2toR5totals,behaviorsmostcommonlyobservedwerebeinganactivefollower(seeTable44)andempathiclistening(seeTable46).Thesewerefollowedbyemotionallyintelligentbehaviors(seeTable45)andbeingtransparentincommunications(seeTable49).Thenexthighesttotalswerefortheabilitytodisagreeproductively(seeTable50),beingmutuallyrespectful(seeTable47),creatingacultureofmutualaccountability(seeTable52),andmovingtowardsasolution(seeTable48).Asmentionedabove,therewasacomparisonoftreatmentandcontrolgroupsinthefrequencyandpatternoftheelevenbehaviorsfromR2throughR5.Cross-tabulationswererunlookingatthepercentageofindividualsinthecontrolandtreatmentgroupsengagingineachspecificbehavior,acrossweeks2through5.Thistypeofanalysiswaschosensinceitallowedadirectcomparisonbetweenthetwogroups,toseewhetherornotbeinginonegroupincreasedordecreasedthelikelihoodofengaginginthatparticularbehavior.Theeightthatshowedastatisticallysignificantdifferenceincludebehaviorsrelatedtosharingleadership,trustinmovingtowardsasolution,beingtransparentincommunication,emotionallyintelligentbehavior,theabilitytodisagreeproductively,definingacommonpurpose,creatingacultureofmutualaccountability,andproductivelymanagingdisruption.
ComparisonBetweenControlandTreatmentbyBehaviorSharesLeadershipTheoccurrenceofthebehaviorwashighestinR3,followedbyR2andR5,thenR4.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthesebehaviorsovertimeincomparingcontrolandtreatmentgroups.WhilethecontrolgrouphadmoreoccurrencesthisbehaviorinR2comparedtothetreatmentgroup,andtheywereroughlyeveninR4,thetreatmentgrouphadroughlytwiceasmanyoccurrencesinR3andR5,comparedtothecontrolgroup.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 81
Table43:SumofSharesLeadershipBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 6 10 3 112(treatment) 10 15 6 63(treatment) 3 8 5 94(treatment) 3 10 4 105(control) 10 7 5 86(control) 10 7 6 27(control) 8 1 3 68(control) 2 9 2 2Overallsum 52 67 34 54Figure2:SharesLeadership,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
ActivefollowerThebehaviorofactivefollowingwasoneofthemorecommonbehaviorsrecorded,withtotalsofover200instancesobservedforeachweekfromR2throughR5;thehighestfrequencieswereinR2andR4.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthesebehaviorsovertimeincomparingcontrolandtreatmentgroups.Table44:SumofActiveFollowerBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 19 26 37 312(treatment) 32 31 24 403(treatment) 32 33 31 294(treatment) 32 28 26 355(control) 48 34 35 366(control) 39 27 27 337(control) 36 28 25 348(control) 45 24 20 35Overallsum 283 231 225 273
22
43
18
36
30
2416
18
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 82
EmotionallyintelligentbehaviorTherewereahighernumberofoccurrencesinR3andR5,comparedtoR2andR4.Therewasastatisticallysignificantdifferenceforthefrequencyandpatternofemotionallyintelligentbehaviorobservedincomparingthecontrolandtreatmentgroups.WhilethecontrolgrouphadaslightlyhighernumberofoccurrencesinR2,thetreatmentgrouphadahighernumberthanthecontrolgroupforR3,R4andR5.Table45:SumofEmotionallyIntelligentBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 10 20 16 222(treatment) 13 26 14 193(treatment) 10 12 16 234(treatment) 8 17 12 235(control) 16 22 20 206(control) 16 13 11 177(control) 8 14 7 98(control) 13 12 5 7Overallsum 94 136 101 140Figure3:EmotionallyIntelligentBehavior,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
41
7558
87
53
61
43
53
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 83
EmpathiclisteningFromR2throughR5therewasasteadyincreaseinempathiclisteningacrossallfoursessions.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthesebehaviorsovertimeincomparingcontrolandtreatmentgroups.Table46:SumofEmpathicListeningBehaviorsObserved,HighSchoolStudentsTeam R2 R3 R4 R51(treatment) 12 23 35 302(treatment) 13 26 24 323(treatment) 23 20 24 334(treatment) 14 25 25 325(control) 35 24 31 376(control) 26 21 27 317(control) 23 21 23 298(control) 31 22 18 26Overallsum 177 182 207 250MutualrespectThetotalinstancesofmutualrespectobservedweresimilaracrossalleightteamsinR2andR3,dippedinR4andthenshotupsubstantiallyinR5.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthesebehaviorsovertimeincomparingcontrolandtreatmentgroups.Table47:SumofMutualRespectBehaviorsObserved,HighSchoolStudentsTeam R2 R3 R4 R51(treatment) 5 11 0 192(treatment) 6 14 7 173(treatment) 18 8 9 174(treatment) 3 11 0 175(control) 9 6 3 126(control) 6 4 1 177(control) 7 4 0 128(control) 7 7 3 6Overallsum 61 65 23 117
AudienceViewpointsConsultingArtofScienceLearningResearchReport 84
TrustinmovingtowardsolutionThebehaviorsobservedfortrustinmovingtowardasolutionincreasedsubstantiallyfromR2toR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthesebehaviorsbetweencontrolandtreatmentgroups.Thetreatmentgroupshowedahighernumberofthesebehaviorseachofthefourweeks,withthelargestdifferencesbeinginR3andR4.Table48:SumofTrustinMovingTowardSolutionBehaviorsObserved,HighSchoolStudentsTeam R2 R3 R4 R51(treatment) 0 14 5 212(treatment) 4 10 10 173(treatment) 4 11 12 164(treatment) 0 0 6 145(control) 6 7 8 226(control) 0 6 4 187(control) 6 3 4 98(control) 2 3 4 13Overallsum 22 54 53 130Figure4:TrustinMovingTowardSolution,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
14
35 33
68
819 20
62
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 85
TransparentincommunicationForthisbehavior,therewasasteadyincreasefromR2throughR5.Therewasastatisticallysignificantdifferencebetweencontrolandtreatmentgroupsinthefrequencyandpatternofthesebehaviors.ThenumbersforeachgroupwerealmostidenticalforR2andR5,whilethetreatmentgrouphadmoreoccurrencesinR3andR4.Table49:SumofTransparentinCommunicationBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 4 13 11 142(treatment) 9 16 16 173(treatment) 6 7 15 204(treatment) 4 10 13 165(control) 7 8 12 186(control) 1 8 10 157(control) 9 5 6 168(control) 5 7 7 18Overallsum 45 74 90 134Figure5:TransparentinCommunication,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
23
4655
67
2228
35
67
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 86
AbilitytodisagreeproductivelyParticipants’abilitytodisagreeproductivelyevolvedinsomewhatofabellcurve,withthelargernumberofoccurrencesinR3andR4,andfewerinR2andR5.Therewasastatisticallysignificantdifferencebetweencontrolandtreatmentgroupsinthefrequencyandpatternofthesebehaviors,withthetreatmentgrouphavingmoreoccurrencescomparedtothecontrolgroupinthreeofthefoursessions.Sinceweonlyobservedproductivedisagreement,itisnotpossibletoinferwhethertherewasadifferenceinnon-productivedisagreement.Table50:SumofAbilitytoDisagreeProductivelyBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 3 15 9 82(treatment) 11 20 10 93(treatment) 6 3 16 104(treatment) 8 14 17 65(control) 14 15 16 36(control) 7 8 9 47(control) 6 0 4 58(control) 6 9 5 5Overallsum 61 84 86 50Figure6:AbilitytoDisagreeProductively,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
28
5252
165
32
32 34
159
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 87
DefiningacommonpurposeThehighestlevelofoccurrencewasinR5,followedbyR3,R4andR2.Therewasstatisticallysignificantdifferenceinthefrequencyandpatternofthisbehaviorbetweencontrolandtreatmentgroups.WhileoccurrenceswererelativelysimilarinR2andR4,therewasamuchhigheroccurrenceofthisbehaviorforthetreatmentgroupinR3andR5.Table51:SumofDefiningaCommonPurposeBehaviorsObserved,HighSchoolStudentsTeam R2 R3 R4 R51(treatment) 2 10 7 152(treatment) 10 8 6 93(treatment) 8 10 9 104(treatment) 1 5 3 105(control) 7 7 3 106(control) 7 8 5 57(control) 6 1 8 98(control) 0 6 5 4Overallsum 41 55 46 72Figure7:DefiningaCommonPurpose,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
21
3325
44
20 22 21
28
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 88
CreatingcultureofmutualaccountabilityThebehaviorofcreatingacultureofmutualaccountabilityhadoneofthelargestincreasesfromR2toR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.Thisdifference,however,wasdrivenbyasingleweek(R4)wherethecontrolgrouphadahigherdegreeofthisbehaviorcomparedtothetreatmentgroup.ThetreatmentgrouphadahigherdegreeofthisbehaviorcomparedtothecontrolgroupinR2,R3andR5.Table52:SumofCreatingCultureofMutualAccountabilityBehaviorsObserved,HighSchoolStudentsTeam R2 R3 R4 R51(treatment) 0 6 12 252(treatment) 0 9 0 163(treatment) 7 9 11 164(treatment) 0 4 4 115(control) 4 10 22 196(control) 1 6 7 227(control) 1 3 5 88(control) 0 8 3 14Overallsum 13 55 64 131Figure8:CreatingaCultureofMutualAccountability,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
7
28
27
68
627
3763
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 89
ProductivelymanagesdisruptionThehighestnumberofoccurrenceswasobservedinR2andR4,withlowernumbersforR3andR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups,thewithcontrolgroupshowingthestrongerperformance.ThetreatmentgrouphadahighernumberofoccurrencesinR2,duetoalargenumberofoccurrencesinoneparticulartreatmentteam,whilethecontrolgrouphadahighernumberofoccurrencesinR3toR5.Table53:SumofProductivelyManagesDisruptionBehaviorsObserved,HighSchoolStudents Team R2 R3 R4 R51(treatment) 0 1 7 32(treatment) 13 6 3 63(treatment) 5 4 9 14(treatment) 5 1 7 25(control) 5 5 10 86(control) 3 5 9 47(control) 8 2 6 08(control) 2 2 5 1Overallsum 41 26 56 25Figure9:ProductivelyManagesDisruption,SumofObservedBehaviorsinGroupsOverTime,HighSchoolStudents
ComparisonBetweenControlandTreatmentintheFinalSessionLookingatteams’observedbehaviorsoverthecourseoftheprojectallowedforanunderstandingofhowtheteamsprogressedduringtheweeklysessions.Thefinalsession,R5representstheculminationoftheproject’sinnovationtraining,andlookingspecificallyattheR5outcomesgivesustheabilitytostudythewaysinwhichthecollaborativebehaviorsofthecontrolandtreatmentgroupswereimpactedbythe20hoursoftraining.Inordertocomparethecomparativeimpactsoffullthefull20-hourintervention,astatisticaltestwasrun,lookingatthedifferenceinthefrequencyofeachoftheelevenbehaviorsinR5,betweenthecontrolandtreatmentgroups.
23
12
26
12
1814
30
13
R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 90
Oftheelevencomparisons,thefollowingtableshowsfivebehaviorswithstatisticallysignificantdifferences:sharesleadership,emotionallyintelligentbehaviors,mutualrespect,abilitytodisagreeproductively,anddefiningacommonpurpose(seeTable54).Alloftheseshowedahigherlevelofoccurrenceforthetreatmentgroup.BothcontrolandtreatmentteamsspentR5finalizingtheirfinalpresentationsandbusinesscasesrelatedtothechallenge,sotherewerealotofopportunitiesforcollaborativebehaviors.Thetreatmentgroupengagedinthesetypesofbehaviorssignificantlymorethanthetreatmentgroup,suggestingthatthetreatmentgroupwasactingmorecollaborativelyastheyfiguredoutthefinalpiecesanddetailsoftheirpresentations.Table54:SummaryTableofObservedCollaborationandOtherInnovationProcessesDifferences,OnlyforWeek5(R5)(Pre/PostObservationalDifferenceScores) HighSchoolStudentsObservationalCategory
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
Sharesleadership Yes TreatmentActivefollower No -Emotionallyintelligentbehavior
Yes Treatment
Empathiclistening No -Mutualrespect Yes TreatmentTrustinmovingtowardsolution
No -
Transparentincommunication No -
Abilitytodisagreeproductively
Yes Treatment
Definingacommonpurpose Yes Treatment
Creatingcultureofmutualaccountability
No -
Productivelymanagesdisruption
No -
AudienceViewpointsConsultingArtofScienceLearningResearchReport 91
Self-ReportedTeamCollaborationRatingsofHighSchoolStudentsThetablesinthissectionaretheindividualself-reportedteamratingsofthehighschoolparticipantsoftheirownbehaviorsduringeachsession(seeTable55throughTable65).WhiletheobservedbehaviorsdidnotincludeR1,eachparticipantfilledoutthesurveyattheendofeachsession,includingR1.Therefore,comparisonsforself-reportedsessionsaremadebetweenR1andR5.AllofthemeasuressawanincreaseintheaverageratingfromR1toR5.Thethreebiggestincreaseswereanincreaseof1.3foractivefollowing(seeTable56),1.1forempathiclistening(seeTable58),and1.0forabilitytodisputeproductively(seeTable62).Thehighestratingsinthelastweek(R5)includedsharesleadership(seeTable55)andbeinganactivefollower(seeTable56),followedbymutualrespect(seeTable59),andtheabilitytodisagreeproductively(seeTable62).Nextwereempathiclistening(seeTable58),trustinmovingtowardsasolution(seeTable60),definingacommonpurpose(seeTable63),andcreatingacultureofmutualaccountability(seeTable64).R5averagescoresrangedfrom6.0to6.8.NosignificantdifferencesinR1/R5changewereseenbetweencontrolandtreatmentgroups.Table55:SharesLeadershipAverageofSelf-ReportedTeamRatings,HighSchoolStudents Team R1 R2 R3 R4 R51(treatment) 5.5 6.0 5.8 6.8 7.02(treatment) 6.1 6.1 6.3 6.6 6.83(treatment) 6.3 6.1 6.4 5.2 6.04(treatment) 5.1 6.3 6.1 6.5 6.65(control) 3.9 6.0 5.8 4.4 6.26(control) 5.2 6.3 6.6 6.3 6.77(control) 4.7 5.3 5.9 6.1 6.18(control) 5.6 6.8 6.7 6.6 6.6Overallaverage 5.3 6.1 6.2 6.1 6.5Note:Scalewasfrom1(Notatall)to7(Completely)Table56:ActiveFollowerAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.3 6.1 6.0 6.9 6.62(treatment) 6.5 5.5 5.4 6.5 6.63(treatment) 6.3 6.0 6.6 5.2 6.14(treatment) 3.8 6.4 6.0 6.5 6.55(control) 3.4 5.9 5.8 4.4 6.16(control) 5.4 6.3 6.3 6.5 6.87(control) 5.3 5.1 6.4 5.9 6.28(control) 5.8 6.6 6.6 6.7 6.8Overallaverage 5.2 6.0 6.1 6.0 6.5Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 92
Table57:EmotionallyIntelligentBehaviorAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.0 5.9 4.8 6.6 6.52(treatment) 6.1 5.4 6.8 6.3 6.53(treatment) 6.1 6.3 6.1 5.2 4.64(treatment) 5.1 6.0 5.0 6.3 6.35(control) 5.2 5.8 5.8 3.8 6.26(control) 5.3 6.2 6.1 6.0 6.77(control) 5.0 4.9 6.4 6.1 6.38(control) 4.9 6.6 6.7 6.6 6.6Overallaverage 5.3 5.9 6.0 5.8 6.2Note:Scalewasfrom1(Notatall)to7(Completely)Table58:EmpathicListeningAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 4.9 5.1 4.1 6.7 6.92(treatment) 6.3 5.6 6.1 6.6 6.83(treatment) 6.2 6.3 6.5 6.2 5.34(treatment) 5.0 6.1 5.4 6.3 5.85(control) 3.6 5.7 6.0 3.8 6.16(control) 5.6 6.2 6.1 6.2 6.37(control) 5.2 5.4 6.6 6.3 6.38(control) 5.3 6.7 6.6 6.6 6.5Overallaverage 5.2 5.9 5.9 6.1 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table59:MutualRespectAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.9 5.1 4.4 6.7 6.92(treatment) 6.9 6.5 4.8 6.4 6.13(treatment) 6.4 6.9 6.5 5.2 6.24(treatment) 5.6 6.2 6.0 6.4 6.05(control) 4.8 5.3 5.4 4.4 6.86(control) 5.5 6.6 6.2 6.6 6.47(control) 5.8 5.6 6.4 6.0 6.38(control) 4.7 6.9 6.7 5.9 6.8Overallaverage 5.7 6.2 5.8 5.9 6.4Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 93
Table60:TrustinMovingTowardSolutionAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.7 5.4 4.4 6.8 6.52(treatment) 6.1 5.8 6.8 6.5 6.63(treatment) 6.9 6.0 6.4 5.6 5.44(treatment) 5.4 5.4 6.0 6.5 6.15(control) 3.7 5.9 5.1 4.3 6.76(control) 5.3 6.5 6.6 6.6 6.47(control) 5.4 5.1 5.8 6.4 6.28(control) 5.6 6.8 6.7 6.8 6.4Overallaverage 5.5 5.9 6.0 6.1 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table61:TransparentinCommunicationAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.1 5.3 4.8 6.4 6.82(treatment) 6.4 6.8 5.7 6.3 6.73(treatment) 6.1 6.1 6.4 5.1 5.64(treatment) 4.8 6.4 6.0 6.1 6.15(control) 5.3 5.6 6.0 4.9 5.46(control) 5.4 5.9 6.3 6.0 6.07(control) 5.3 5.3 5.7 6.1 6.18(control) 6.0 6.8 6.7 6.6 6.6Overallaverage 5.5 6.0 5.9 5.9 6.2Note:Scalewasfrom1(Notatall)to7(Completely)Table62:AbilitytoDisagreeProductivelyAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.5 4.1 5.8 6.4 6.02(treatment) 6.4 6.0 5.8 6.8 6.13(treatment) 6.1 6.0 6.5 4.7 6.84(treatment) 5.5 5.9 5.4 6.4 6.55(control) 4.4 5.7 5.7 4.1 6.46(control) 5.5 6.1 6.1 6.7 6.47(control) 5.2 5.3 5.8 6.1 6.48(control) 5.2 6.0 6.4 6.6 6.5Overallaverage 5.4 5.6 5.9 5.9 6.4Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 94
Table63:DefiningaCommonPurposeAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.4 5.6 5.6 6.7 6.82(treatment) 6.4 5.9 5.6 5.5 6.53(treatment) 6.2 5.7 6.6 5.0 4.74(treatment) 5.4 5.9 6.0 6.9 6.45(control) 4.6 5.2 5.2 3.7 6.46(control) 5.7 6.5 6.1 6.4 6.77(control) 5.1 4.9 5.8 6.3 6.38(control) 5.7 6.4 6.7 6.6 6.8Overallaverage 5.5 5.8 5.9 5.8 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table64:DefiningaCultureofMutualAccountabilitySelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 5.5 5.9 3.8 6.9 7.02(treatment) 6.1 5.6 5.6 6.6 6.63(treatment) 6.1 5.4 6.3 5.1 5.34(treatment) 5.6 6.3 6.4 6.5 6.65(control) 4.4 5.4 5.4 3.2 5.46(control) 4.7 6.5 6.2 6.1 6.77(control) 5.4 5.1 5.9 5.6 6.38(control) 5.1 6.7 6.7 6.6 6.8Overallaverage 5.3 5.9 5.8 5.8 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table65:ProductivelyManagedDisruptionAverageofSelf-ReportedTeamRatings,HighSchoolStudentsTeam R1 R2 R3 R4 R51(treatment) 3.9 3.5 3.6 5.0 6.12(treatment) 6.0 5.5 5.3 5.8 6.43(treatment) 5.7 5.3 6.6 5.9 6.04(treatment) 5.5 6.1 5.4 5.9 6.45(control) 4.5 6.1 4.7 4.2 6.46(control) 4.3 5.5 6.7 6.2 5.97(control) 4.8 4.9 5.7 5.9 6.38(control) 4.5 5.2 5.7 5.7 4.3Overallaverage 4.8 5.3 5.5 5.6 6.0Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport 95
HighSchoolStudentBehaviorFindingsSummaryTherewereparticipationeffectsforbothcontrolandtreatmentgroupsofhighschoolstudents.Thiswasespeciallytruefortheobservedbehaviors,whichshowedstatisticallysignificantincreasesovertheweeklysessionsforboththecontrolgroupsandtreatmentgroups.Intermsofthebehaviorsimpacted,itisinterestingtonotethatthecontrolgroupshowedincreasesoverthefourweeksobservedinevenmoreofthebehaviors(6ofthe11)thanthetreatmentgroup(5ofthe11).Whenlookingattheself-reportedteamratingsofthesamemeasures,oneseesadifferentpicture.Onlytwoofthebehaviors,mutualrespectandtrustinmovingtowardsasolution,showedsignificantdifferencesbetweenthetwogroups;inbothcases,itwasthecontrolgroupthatgaveitselfhigherratingsfromR1toR5.Whencomparingthetwogroupsdirectly,therewere8ofthe11behaviorswherethefrequencyandpatternsofthebehaviorsdifferedsignificantlybetweencontrolandtreatment.In6ofthese8behaviors,thetreatmentgroupshowedthestrongerperformance.Thesebehaviorsweresharesleadership,trustinmovingtowardasolution,transparentincommunication,emotionallyintelligentbehavior,disagreeproductivelyanddefiningacommonpurpose.Intheremaining2ofthese8behaviors(creatingacultureofmutualaccountabilityandproductivelymanagesdisruption)thecontrolgroupshowedmarginally,butstatisticallysignificant,strongerperformance.TheR5comparisonallowedforacomparisonbetweenthecontrolandtreatmentgroupsduringtheirlastsessions,whenparticipantswerecompletingtheircourseofstudyandteamswerefinishingtheirworkonthechallengeandmakingalloftheirfinaldecisionswithrespecttobusinesscasesandpresentations.Thus,R5datagiveusasenseofthecumulativeimpactofthefulltwenty-hourinterventiononcollaborativebehaviorofcontrolandtreatmentgroups;ifbehaviordifferenceswouldbeexpectedtoexistanywherebetweenthetwogroups,itwouldbeduringthislastsessionattheculminationoftheproject.StatisticallysignificantbehavioraldifferencesforfiveoftheelevenbehaviorswereobservedinR5:sharesleadership,beingemotionallyintelligent,mutualrespect,theabilitytodisagreeproductively,anddefiningacommonpurpose.Forallfive,thetreatmentgrouphadasignificantlyhigheroccurrenceofthesebehaviors.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 96
EarlyCareerSTEMProfessionalBehaviorFindingsThetablesinthefirstpartofthissection(Table66throughTable76)showtheobservedbehaviorratingsbytheresearchersfromR1throughR5;seeAppendixCfortheindividualbehaviorsstudied.Atableisincludedforeachoftheelevenbehaviorsobservedbytheresearcher,breakingdownthenumberoftimesthisparticularbehaviorwasobservedforeachteamduringeachweek.Theoverallsumofbehaviorsobservedineachweekisincludedatthebottomofthetable.Inordertocomparethegroupsoneachbehavior,statisticaltestswererunonthedifferencesbetweenthefrequencyandpatternsofoccurrencethesebehaviorswereobservedincontrolandtreatmentovertheR1toR5period.Iftherewasastatisticallysignificantdifferencebetweenthecontrolandtreatmentgroups,alinegraphwasincludedbelowthetable.Tableswithoutaccompanyinglinegraphsshowednostatisticallysignificantdifferencesbetweencontrolandtreatment.BasedontheR1toR5totals,behaviorsmostcommonlyobservedwerebeinganactivefollower(seeTable67)andmutualrespect(seeTable70).Thesewerefollowedbyempathiclistening(seeTable69)andsharingleadership(seeTable66).Thenexthighesttotalswereforemotionallyintelligentbehavior(seeTable68),definingacommonpurpose(seeTable74),andtrustinmovingtowardsasolution(seeTable71).Asmentionedabove,therewasacomparisonoftreatmentandcontrolgroupsinthefrequencyandpatternoftheelevenbehaviorsfromR1toR5.Cross-tabulationswererunlookingatthepercentageofindividualsinthecontrolandtreatmentgroupsengagingineachspecificbehavior,acrossweeks1through5.Thistypeofanalysiswaschosensinceitallowedadirectcomparisonbetweenthetwogroups,toseewhetherornotbeinginonegroupincreasedordecreasedthelikelihoodofengaginginthatparticularbehavior.Theseventhatshowedastatisticallysignificantdifferenceincludebehaviorsrelatedtosharingleadership,beinganactivefollower,emotionallyintelligentbehavior,empathiclistening,mutualrespect,trustinmovingtowardsasolution,andbeingtransparentincommunication.
ComparisonBetweenControlandTreatmentbyBehaviorSharesLeadershipTherewerefeweroccurrencesofthebehaviorinR1andR2,thenhigheroccurrencesinR3throughR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.ThecontrolandtreatmentgroupswereverysimilarinR1,R2,andR4,thecontrolgroupwashigherinR3andthetreatmentgroupwashigherinR5.ItisinterestingtonotethatwhilethetreatmentgroupincreasessteadilyeachsessionfromR1toR5,thecontrolgrouppeaksatR4andthendeclinesinR5.
AudienceViewpointsConsultingArtofScienceLearningResearchReport 97
Table66:SumofSharesLeadershipBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 7 8 10 27 242(treatment) 2 4 18 22 283(treatment) 1 1 14 20 294(treatment) 1 5 14 31 275(control) 0 5 21 26 246(control) 6 4 26 32 267(control) 4 3 28 24 228(control) 2 8 16 18 15Overallsum 23 38 147 200 195Figure10:SharesLeadership,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
11
2056
100 108
12
18
91
10087
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 98
ActivefollowerOccurrencesofthisbehaviorincreasedinfrequencyeachweekbetweenR1andR3(particularlyinR3)andthenessentiallyplateauedduringtheremainingsessions.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.ThetreatmentgroupwashigherthanthecontrolgroupinR1andR2,thecontrolandtreatmentgroupswereverysimilarinR3andR4,whilethetreatmentgroupwashigherthanthecontrolgroupinR5.Table67:SumofActiveFollowerBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 22 22 38 44 502(treatment) 12 17 53 34 493(treatment) 20 15 40 35 524(treatment) 15 20 33 50 525(control) 7 13 37 40 356(control) 8 15 47 40 357(control) 14 13 50 44 418(control) 5 11 34 31 29Overallsum 103 126 332 318 343Figure11:ActiveFollower,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
69 74164
163203
34 52
168
155 140
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport 99
EmotionallyintelligentbehaviorTherewereloweroccurrencesforthebehaviorinR1andR2,alargeincreaseforR3,aslightdecreaseinR4,thenanotherincreaseinR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.WhiletherewasahigheroccurrenceforthecontrolgroupinR2,theotherweekshadverysimilaroccurrencesforthecontrolandtreatmentgroups.However,acrossthefivesessions,thetreatmentgrouphadagreaterfrequencyofthisbehavior.Table68:SumofEmotionallyIntelligentBehaviorObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 2 3 10 20 122(treatment) 1 3 12 5 193(treatment) 3 0 11 6 114(treatment) 1 4 14 8 125(control) 2 6 6 13 186(control) 2 5 15 8 147(control) 1 1 12 12 98(control) 1 5 14 4 10Overallsum 13 27 94 76 105Figure12:EmotionallyIntelligentBehavior,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
7
10
4739
54
6
17
47
37
51
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport100
EmpathiclisteningTherewasasteadyincreaseofoccurrencesofthebehaviorfromR1toR5,withtentimesthenumberofoccurrencescomparingR1toR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.TherewasaverysimilarnumberofoccurrencesforthecontrolandtreatmentgroupsforR1andR2,higheroccurrencesforthecontrolgroupinR3andR4,thenahigheroccurrenceforthetreatmentgroupinR5.ItisinterestingtonotethatwhilethetreatmentgroupincreasessteadilyeachsessionfromR1toR5,thecontrolgroupplateausatR3andR4,andthendeclinesinR5.Table69:SumofEmpathicListeningBehaviorObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 3 11 11 21 282(treatment) 3 9 17 13 403(treatment) 2 8 14 17 284(treatment) 2 10 8 24 305(control) 2 11 14 22 216(control) 3 7 29 26 197(control) 2 7 24 31 218(control) 3 10 21 16 20Overallsum 20 73 138 170 207Figure13:EmpathicListening,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
10
38
50
75
126
1035
8895
81
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport101
MutualrespectTherewereloweroccurrencesforthebehaviorinR1andR2,thenmuchhighernumbersofoccurrencesinR3throughR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.WhilethenumberofoccurrenceswassimilarfromR1throughR4,therewasahighernumberofoccurrencesforthetreatmentgroupinR5.Table70:SumofMutualRespectBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 5 13 36 45 542(treatment) 2 4 46 35 543(treatment) 1 3 43 36 544(treatment) 1 9 38 50 505(control) 3 8 39 40 396(control) 3 7 48 40 407(control) 5 9 54 45 458(control) 0 11 35 27 35Overallsum 20 64 339 318 371Figure14:MutualRespect,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
9 29
163
166
212
1135
176
152159
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport102
TrustinmovingtowardsolutionTherewerelowernumberofoccurrencesofthebehaviorinR1andR2,anincreaseforR3andR4,thenanotherincreaseforR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.ThecontrolandtreatmentgroupsweresimilarforR1andR2,thecontrolgroupwasmodestlyhigherinR3,whilethetreatmentgroupwashigherinR4andmuchhigherinR5.Thetreatmentgrouproseeachweek,whilethecontrolgroupdeclinedconsiderablybetweenR3andR4.Table71:SumofTrustinMovingTowardSolutionBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 3 7 8 9 222(treatment) 3 2 9 7 243(treatment) 1 1 9 9 104(treatment) 0 5 7 18 255(control) 0 5 12 6 36(control) 1 4 8 1 137(control) 2 4 14 12 108(control) 0 5 10 7 11Overallsum 10 33 77 69 118Figure15:TrustinMovingTowardSolution,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
715
33
43
81
3
18
44
2637
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport103
TransparentincommunicationTherewererelativelyfewoccurrencesofthebehaviorinR1toR3,withsubstantialincreasesinbothR4andR5.Therewasastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.TherewererelativesimilaroccurrencesforthecontrolandtreatmentgroupsforR1toR3,thenhigheroccurrencesforthetreatmentgroupforR4andR5.Table72:SumofTransparentinCommunicationBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 0 0 0 11 132(treatment) 0 2 0 8 113(treatment) 0 1 1 3 84(treatment) 0 3 2 13 215(control) 0 1 1 5 96(control) 2 0 2 5 117(control) 1 1 2 5 148(control) 0 0 1 9 7Overallsum 3 8 9 59 94Figure16:TransparentinCommunication,SumofObservedBehaviorsinGroupsOverTime,EarlyCareerSTEMProfessionals
0
6
3
35
53
3 26
24
41
R1 R2 R3 R4 R5
Treatment Control
AudienceViewpointsConsultingArtofScienceLearningResearchReport104
AbilitytodisagreeproductivelyTherewasasteadyincreaseintheoccurrenceofbehaviorsfromR1toR5.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.Table73:SumofAbilitytoDisagreeProductivelyBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 0 3 3 7 42(treatment) 0 0 2 3 93(treatment) 0 2 0 3 04(treatment) 0 2 0 7 105(control) 0 1 3 3 76(control) 0 1 8 7 07(control) 1 2 3 5 58(control) 1 0 5 3 5Overallsum 2 11 24 38 40DefiningacommonpurposeTherewasasteadyincreaseintheoccurrenceofbehaviorsfromR1toR5.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.Table74:SumofDefiningaCommonPurposeBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 4 3 3 14 162(treatment) 1 3 13 6 113(treatment) 4 3 5 8 124(treatment) 2 7 5 10 165(control) 2 6 9 7 86(control) 3 3 16 16 117(control) 2 3 11 14 148(control) 1 10 10 9 10Overallsum 19 38 72 84 98
AudienceViewpointsConsultingArtofScienceLearningResearchReport105
CreatingcultureofmutualaccountabilityTherewererelativelylowoccurrencesofthebehaviorfromR1toR5.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.Table75:SumofCreatingCultureofMutualAccountabilityBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 0 0 0 2 12(treatment) 0 0 1 1 03(treatment) 0 0 0 2 04(treatment) 0 0 0 4 05(control) 0 1 1 1 06(control) 0 0 2 2 37(control) 0 0 0 2 18(control) 0 1 3 0 0Overallsum 0 2 7 14 5ProductivelymanagesdisruptionTherewerelessfrequentoccurrencesofthebehaviorinR1andR2,anincreaseinR3andR4,thenadecreaseinR5.Therewasnotastatisticallysignificantdifferenceinthefrequencyandpatternofthebehaviorbetweencontrolandtreatmentgroups.Table76:SumofProductivelyManagesDisruptionBehaviorsObserved,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 2 1 0 9 42(treatment) 0 0 0 8 43(treatment) 0 0 0 5 34(treatment) 1 0 0 5 35(control) 0 0 4 8 46(control) 2 0 8 9 37(control) 2 0 5 8 48(control) 0 2 6 2 3Overallsum 7 3 23 54 28
ComparisonbetweenControlandTreatmentintheFinalSessionLookingatteams’behaviorsoverthecourseoftheprojectallowedforanunderstandingofhowtheteamsprogressedduringtheweeklysessions.Thefinalsession,R5representstheculminationoftheproject’sinnovationtraining,andlookingspecificallyattheR5outcomesgivesustheabilitytostudythewaysinwhichthecollaborativebehaviorsofthecontrolandtreatmentgroupswereimpactedbythe20hoursoftraining.Inordertocomparethecomparativeimpactsoffullthefull20-hourintervention,astatisticaltestwasrun,lookingatthedifferenceinthefrequencyofeachoftheelevenbehaviorsinR5,betweenthe
AudienceViewpointsConsultingArtofScienceLearningResearchReport106
controlandtreatmentgroups. Oftheelevencomparisons,thefollowingtableshowstwobehaviorswithstatisticallysignificantdifferences:mutualrespectandtrustmovingtowardasolution(seeTable77).Bothoftheseshowedahigherlevelofoccurrenceforthetreatmentgroup.BothcontrolandtreatmentteamsspentR5finalizingtheirfinalpresentationsandbusinesscasesrelatedtothechallenge,sotherewerealotofopportunitiesforcollaborativebehaviors.Thetreatmentgroupengagedinthesetypesofbehaviorssignificantlymorethanthetreatmentgroup,suggestingthatthetreatmentgroupwasactingmorecollaborativelyastheyfiguredoutthefinalpiecesanddetailsoftheirpresentations.Table77:SummaryTableofObservedCollaboration,OnlyforWeek5(R5)(Pre/PostObservationalDifferenceScores) EarlyCareerSTEMProfessionalsObservationalCategory
StatisticallySignificantGroupDifferences?
GroupwithBetterPerformance(Pre/Post)
Sharesleadership No -Activefollower No -Emotionallyintelligentbehavior
No -
Empathiclistening No -Mutualrespect Yes TreatmentTrustinmovingtowardsolution
Yes Treatment
Transparentincommunication No -
Abilitytodisagreeproductively
No -
Definingacommonpurpose No -
Creatingcultureofmutualaccountability
No -
Productivelymanagesdisruption
No -
AudienceViewpointsConsultingArtofScienceLearningResearchReport107
Self-ReportedTeamCollaborationRatingsofEarlyCareerSTEMProfessionalsThefollowingtablesaretheindividualself-reportedteamratingsoftheadultparticipantsoftheirownbehaviorsduringeachsession.AllbutoneofthemeasuressawanincreaseintheaverageratingfromR1toR5;sharesleadership(seeTable78)showedneitheranincreasenoradecrease.Thethreebiggestincreaseswereanincreaseof0.9forbeingabletodisagreeanddisputeproductively(seeTable85),anincreaseof0.6fordefiningacommonpurpose(seeTable86),andcreatingacultureofmutualaccountability.Thehighestratingsinthelastweek(R5)includedmutualrespect(seeTable82)andcreatingacultureofmutualaccountability(seeTable87).Thesewerefollowedbybeinganactivefollower(seeTable79),empathiclistening(seeTable81),beingtransparentincommunication(seeTable84),anddefiningacommonpurpose(seeTable86).Table78:SharesLeadershipAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.4 6.6 6.7 6.2 6.02(treatment) 6.0 5.6 6.2 6.0 6.63(treatment) 5.8 5.7 6.1 5.8 6.64(treatment) 6.6 6.4 6.5 6.6 65(control) 6.3 6.6 6.2 6.6 36(control) 6.2 6.9 6.9 5.4 6.97(control) 5.6 6.1 6.0 6.8 6.28(control) 5.4 6.7 6.7 5.4 7.0Overallaverage 6.3 6.3 6.4 6.1 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table79:ActiveFollowerAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.4 6.4 6.0 6.3 5.92(treatment) 6.2 5.3 6.6 5.9 6.43(treatment) 5.8 5.8 6.0 6.1 6.34(treatment) 6.6 6.3 6.5 6.4 6.65(control) 6.5 6.4 6.2 6.7 6.96(control) 5.5 6.1 6.6 5.9 6.27(control) 5.4 5.8 5.7 5.8 6.98(control) 5.9 6.0 6.4 5.9 5.7Overallaverage 6.0 6.0 6.2 6.1 6.4Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport108
Table80:EmotionallyIntelligentBehaviorAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.6 6.2 6.7 6.7 6.02(treatment) 6.6 4.8 6.6 6.1 6.73(treatment) 5.4 5.4 6.0 5.2 6.44(treatment) 6.3 6.0 6.7 6.2 6.05(control) 5.7 6.5 5.9 6.7 6.76(control) 5.9 6.2 6.6 5.4 6.17(control) 6.2 6.1 5.0 6.0 6.68(control) 5.4 5.7 6.3 5.4 5.4Overallaverage 6.0 5.9 6.2 6.0 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table81:EmpathicListeningAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.7 6.3 6.8 6.2 5.82(treatment) 6.7 6.2 6.4 6.4 6.73(treatment) 6.2 5.7 6.3 6.1 6.74(treatment) 6.5 6.4 6.8 6.1 6.25(control) 6.5 6.7 6.0 6.7 6.96(control) 5.9 6.2 6.6 6.0 6.47(control) 5.7 5.4 5.7 6.7 6.98(control) 6.0 6.4 6.7 5.9 5.9Overallaverage 6.3 6.2 6.4 6.3 6.4Note:Scalewasfrom1(Notatall)to7(Completely)Table82:MutualRespectAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.9 6.8 6.8 6.7 6.02(treatment) 6.9 6.3 6.8 6.6 6.93(treatment) 5.6 6.2 6.3 6.1 6.74(treatment) 6.8 6.6 6.9 6.6 6.35(control) 6.7 6.7 6.5 6.7 7.06(control) 6.1 6.7 6.6 5.7 6.57(control) 5.6 5.9 6.6 7.0 7.08(control) 6.0 6.9 6.7 6.4 6.0Overallaverage 6.3 6.5 6.6 6.5 6.5Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport109
Table83:TrustinMovingTowardSolutionAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 5.8 6.3 6.3 6.0 5.42(treatment) 6.3 5.3 6.2 6.3 6.83(treatment) 5.0 5.3 5.4 6.0 6.34(treatment) 6.4 6.5 6.6 6.5 6.05(control) 6.7 6.6 6.4 6.7 7.06(control) 6.4 6.5 6.6 6.0 6.47(control) 6.0 5.3 6.6 7.0 6.98(control) 5.6 6.7 5.9 5.7 5.6Overallaverage 6.0 6.1 6.3 6.3 6.3Note:Scalewasfrom1(Notatall)to7(Completely)Table84:TransparentinCommunicationAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.7 6.7 5.8 6.8 6.12(treatment) 6.4 6.0 6.7 6.2 6.73(treatment) 6.0 5.8 6.1 5.9 6.64(treatment) 6.5 6.3 6.6 6.5 6.85(control) 6.5 6.7 6.2 6.8 7.06(control) 6.1 6.6 6.7 5.4 6.27(control) 6.1 5.7 6.7 6.8 6.88(control) 5.3 6.6 6.4 6.0 5.1Overallaverage 6.2 6.3 6.4 6.3 6.4Note:Scalewasfrom1(Notatall)to7(Completely)Table85:AbilitytoDisagreeProductivelyAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.6 5.2 6.6 6.8 6.02(treatment) 5.2 6.0 5.9 5.1 6.43(treatment) 3.9 4.0 5.7 5.3 6.04(treatment) 6.4 5.5 6.6 5.9 5.85(control) 4.3 6.7 5.9 6.0 7.06(control) 4.9 5.9 5.7 5.1 6.47(control) 4.4 4.9 5.6 5.2 5.88(control) 5.3 6.4 5.6 5.9 5.9Overallaverage 5.2 5.5 6.0 5.7 6.1Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport110
Table86:DefiningaCommonPurposeAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 5.8 6.1 5.8 6.4 6.12(treatment) 6.1 5.6 6.8 6.1 6.93(treatment) 6.0 5.8 5.9 6.0 5.94(treatment) 6.3 6.1 6.5 6.6 6.35(control) 6.2 6.6 6.0 6.6 7.06(control) 5.7 6.0 6.1 5.9 6.57(control) 5.6 5.1 5.6 5.8 6.98(control) 5.0 6.4 6.3 4.9 5.6Overallaverage 5.8 5.9 6.1 6.1 6.4Note:Scalewasfrom1(Notatall)to7(Completely)Table87:CreatingCultureofMutualAccountabilityAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 6.3 5.4 6.8 6.6 6.02(treatment) 6.3 6.0 6.3 6.2 6.93(treatment) 4.6 5.7 6.0 6.0 6.44(treatment) 6.0 6.0 6.4 6.4 6.55(control) 6.2 6.5 5.7 6.5 7.06(control) 5.2 6.1 6.7 5.9 6.57(control) 5.8 5.9 5.9 4.8 6.98(control) 4.4 6.3 6.6 6.0 5.3Overallaverage 5.6 6.0 6.3 6.0 6.5Note:Scalewasfrom1(Notatall)to7(Completely)Table88:ProductivelyManagesDisruptionAverageofSelf-ReportedTeamRatings,EarlyCareerSTEMProfessionalsTeam R1 R2 R3 R4 R51(treatment) 2.9 3.4 4.4 5.2 4.62(treatment) 3.1 4.9 5.7 5.7 5.63(treatment) 3.1 3.9 3.6 4.6 5.44(treatment) 4.0 3.9 4.0 4.5 5.05(control) 3.3 5.9 5.6 5.9 7.06(control) 4.9 4.6 5.4 4.9 6.57(control) 3.6 3.7 3.7 4.1 5.38(control) 3.9 5.3 5.7 4.7 4.4Overallaverage 3.6 4.4 4.7 4.9 5.5Note:Scalewasfrom1(Notatall)to7(Completely)
AudienceViewpointsConsultingArtofScienceLearningResearchReport111
EarlyCareerSTEMProfessionalsBehaviorFindingsSummaryFortheearlycareerSTEMprofessionals,thereweredifferencesintheweeklytotalsforthetreatmentgroupsinabouttwothirdsoftheobservationbehaviors,whileonlyonebehaviorforthecontrolgroupshowedastatisticallysignificantdifference.Whencomparingthetwogroupsdirectly,therewere7ofthe11behaviorswherethefrequencyandpatternsofthebehaviorsdifferedsignificantlybetweencontrolandtreatment.In4ofthese7behaviors,thetreatmentgroupshowedtheunambiguouslystrongerperformance.Thesebehaviorswereactivefollower,mutualrespect,trustinmovingtowardasolutionandtransparentincommunication.In2ofthebehaviors,sharingleadershipandempathiclistening,thecontrolgroupshowedamarginally,butstatisticallysignificant,strongerperformance.Inemotionallyintelligentbehavior,thetreatmentgroupshowedamarginally,butstatisticallysignificant,strongerperformance.Fortheself-reportedteambehaviors,therewerenostatisticallysignificantdifferencesforeitherthecontrolortreatmentgroupacrossthefivesessions.TheR5comparisonallowedforacomparisonbetweenthecontrolandtreatmentgroupsduringtheirlastsessions,whenparticipantswerecompletingtheircourseofstudyandteamswerefinishingtheirworkonthechallengeandmakingalloftheirfinaldecisionswithrespecttobusinesscasesandpresentations.Thus,R5datagiveusasenseofthecumulativeimpactofthefulltwenty-hourinterventiononcollaborativebehaviorofcontrolandtreatmentgroups;ifbehaviordifferenceswouldbeexpectedtoexistanywherebetweenthetwogroups,itwouldbeduringthislastsessionattheculminationoftheproject.Statisticallysignificantbehavioraldifferencesfor2ofthe11behaviorswereobservedinR5:mutualrespectandtrustinmovingtowardsasolution.Forbothofthese,thetreatmentgrouphadasignificantlyhigheroccurrenceofthesebehaviors.
AudienceViewpointsConsultingArtofScienceLearningResearchReport112
Hypothesis3Findings:TeamInnovationOutcomesHypothesis 3: Arts-based innovation training, compared to traditional innovation training, enhances the novelty, impact and feasibility of team innovation outcomes. WhileHypothesis1focusedonindividuals,andHypothesis2focusedontheteamastheunitofanalysis,Hypothesis3focusedontheinnovationoutcomesoftheteamsbyanalyzingtheprototypeproducts,processesandservicestheydeveloped.Theanalysiswasperformedbyadistinguishedgroupofexpertpanelists,composedofthreemembersoftheselectioncommitteeoftheProductDevelopmentManagementAssociation(PDMA)’sOutstandingCorporateInnovationAwards,includingthatcommittee’sfoundingandcurrentChairs.Thepanel,consultingwiththeprojectandresearchteams,developedanassessmentrubricidentifyingandweightingsevenmeasurestogaugethequalityoftheinnovationoutcomesincludingteaminnovationoutputsandworkproducts.Theysubsequentlyappliedtherubrictothenewproduct,processandservicesolutionconceptsdevelopedbytheteams.Allscoresarebasedonaratingscalefrom1-5points.Eachofthescaleswasgivenanoverallweight,inordertocalculatetherelativeimportanceofeachforthetotalscore.
1. InsightintoChallenge:Graspoftransportationneeds,conditionsandopportunities.[Overallweight=15%]
2. ClarityandRelevanceofProblem[Overallweight=15%]3. ProblemSolvingStrategy[Overallweight=25%]4. Impact:assessedbyvalueproposition,potentialmarketsizeandpenetration.[Overallweight=15%]5. DevelopmentalStrategy[Overallweight=10%]6. Feasibility:Market,technology,capacity,cost,competition,risk/barriersetc.[Overallweight=10%]7. Teamwork/Collaboration:Integrationofskills;distributedandcollaborativeeffort[Overall
weight=10%]
ThepanelistsmetinpersonovertwodaysattheKelleySchoolofBusinessatIndianaUniversitytodiscussandassesstheproduct,processandserviceinnovationscreatedbytheteamsinresponsetothecivicchallenges(seeAppendicesJandK).Thechallengesincludedtransportationalternativesforthehighschoolstudents,waterresourcesfortheearlycareerSTEMprofessionals.Aheadofthepanelmeeting(butafterthepanel’scompletionofworkontheassessmentrubric),thepanelistsreceivedthebusinesscasescreatedbyeachoftheteamsfortheiradvancereview;panelistswereaskednottodiscussthesematerialswiththeircolleaguesaheadofthemeeting.Duringthecourseofthemeeting,panelistsreviewedthreeadditionalcomponents:thePowerPointpresentationscreatedbyeachteamaboutitsinnovation,pre-recordedvideosofeachteam’s5-minuteconceptpresentation,andpre-recordedvideosofeachteam’sresponsestoastandardizedsetofquestions.Takentogether,theseelements(businesscase,PowerPointpresentations,videoofconceptpresentationandvideoofresponsestoquestions)formedthebasisforthepanel’soverallassessmentofeachteam’sinnovationoutcomes.
AudienceViewpointsConsultingArtofScienceLearningResearchReport113
Theclassroomhostingthepanelmeetingwasequippedwithatwo-screensetup,whereeachPowerPointwasprojectedontheleftscreenwhilethevideoofthatteam’spresentationwasprojectedsynchronouslyontherightscreen,withaudioheardthroughtheclassroom’ssoundsystem.AprojectpartnerstartedthevideoontherightscreenandthenmanuallyranthroughthePowerPointslidesonthelefttomatchwheretheteamwasinthevideopresentation.Inthismanner,thepanelistswereabletobothwatchtheteamspresentandviewtheirPowerPointpresentationsatthesametime,approximatingtheexperienceofbeingintheclassroomwhentheteamsgavetheirpresentations.Thethree-personreviewpanelsattogetherinonerowoftheclassroom,tooknotesandindependentlyfilledoutratingsheetswhileviewingthepresentations.Afterthevideohadfinished,theycompletedtheratingsheetwithouttalkingtoeachother.Whenallthreepanelistsweredonewiththeirindividualratingsforthatteam,theyengagedinadiscussionaroundtheteam’soverallpresentation,itsapproachtotheproblem,theproposedsolution,andothertopicsontheratingsheets.Afterthisgroupdiscussionwasover,thepanelistswentbacktotheirratingsheetsandmadeanyadjustmentsorrescoringofthesheets.Thesheetswerecollectedafterthereviewofeachteam’spresentationwascompleted.Thissameprocesswasrepeatedforeachteam.Eleventeamswerereviewedinthismanneronthefirstdayandfiveonthesecondday.Thereviewersdidnotknowwhichofthe16groupswereinthecontrolortreatmentcondition,andtheorderofconsiderationofteamswithineachgroup(i.e.,adultsandadolescents)wasrandomized.
HighSchoolStudentsFindingsForhighschoolstudents,thetotalweightedteaminnovationoutcomescoresvariedgreatly,rangingfroma1.3toa4.4ona5-pointscale(seeTable89).Next,thescoresofeachitemforeachteaminthecontrolandtreatmentgroupswerecombined,andthetwosetsofcombinedtotalswerecompared(seeTable90).Treatmentoutperformedcontrolonallsevenindividualitemsscored.Fourofthesedifferenceswerestatisticallysignificant:insightintochallenge(2.0difference),clarityandrelevanceoftheproblem(1.7difference),problemsolvingstrategy(1.3difference)andthepotentialimpactoftheirproposal(1.3difference).Whilethedifferencesbetweencontrolandtreatmentontheotherthreeitemsscoreddidnotreachstatisticalsignificance,thetreatmentgroupdidhavehigherratingsthanthecontrolgrouponeach.Similarly,inthetotalweightedteaminnovationoutcomescore,whichusedanaverageweightedtotalscoreacrossallitemsforthecontrolgroupcomparedtotheaverageweightedtotalscoreacrossallitemsforthetreatmentgroup,thetreatmentgrouphadhigherratingsbutthedifferencedidnotreachstatisticalsignificance.
AudienceViewpointsConsultingArtofScienceLearningResearchReport114
Table89:PanelScoresforHighSchoolStudentsTeamFinalProducts(InnovationOutcomes) Control Treatment Team
5Team6
Team7
Team8
Team1
Team2
Team3 Team4
Insightintochallenge 2.8 1.0 1.3 2.3 5.0 1.7 3.0 3.3Clarityandrelevanceofproblem 3.3 1.3 2.7 2.7 5.0 1.7 4.0 3.0
ProblemSolvingStrategy 2.0 1.3 2.5 2.0 4.8 2.0 2.3 2.7
Impact 2.7 1.3 1.7 2.3 4.3 1.7 2.3 3.0
DevelopmentalStrategy 3.0 1.0 3.5 3.8 3.7 1.3 3.3 2.7
Feasibility 2.3 1.3 3.0 3.3 3.3 1.3 4.0 2.0Teamwork/Collaboration 4.0 2.0 5.0 2.7 3.7 2.3 2.7 4.3
Totalweightedscore 2.8 1.3 2.6 2.6 4.4 1.8 3.0 3.0Table90:AveragePanelScoresforHighSchoolStudentsTeamFinalProducts(InnovationOutcomes),ControlandTreatment
Control Treatment
StatisticallySignificantDifference?
Insightintochallenge 1.8 3.8 YesClarityandrelevanceofproblem 2.3 4.0 YesProblemSolvingStrategy 2.0 3.3 YesImpact 1.9 3.2 YesDevelopmentalStrategy 2.5 3.2 NoFeasibility 2.3 3.1 NoTeamwork/Collaboration 3.2 3.6 NoTotalweightedscore 2.3 3.0 No
EarlyCareerSTEMProfessionalsFindingsForearlycareerSTEMprofessionalsthetotalweightedteaminnovationoutcomescoresvariedgreatly,rangingfroma2.2toa4.2ona5-pointscale(seeTable91).Next,thescoresofeachitemforeachteaminthecontrolandtreatmentgroupswerecombined,andthetwosetsofcombinedtotalswerecompared(seeTable92).Whiledifferencesbetweencontrolandtreatmentwerefoundforthehighschoolstudentsontheindividualitems,noneofthedifferencesbetweencontrolandtreatmentonthesevenindividualitemsscoredwerestatisticallysignificantfortheearlycareerSTEMprofessionals.Similarly,therewasnostatisticallysignificantdifferencebetweenthecontrolandtreatmentgroupsinthetotalweightedscoreacrossallitems.
AudienceViewpointsConsultingArtofScienceLearningResearchReport115
Table91:PanelScoresforEarlyCareerSTEMProfessionalsTeamFinalProducts(InnovationOutcomes) Control Treatment Team
5Team6
Team7
Team8
Team1
Team2 Team3
Team4
Insightintochallenge 3.7 3.0 3.7 4.0 3.0 3.7 2.0 4.0
Clarityandrelevanceofproblem 4.0 3.3 3.3 3.7 3.3 3.2 3.0 4.3
ProblemSolvingStrategy 3.8 1.5 3.3 2.3 3.0 2.3 2.0 4.3
Impact 3.2 1.3 2.7 2.2 2.7 2.3 1.7 4.3
DevelopmentalStrategy 2.7 1.6 3.0 2.7 2.0 2.2 1.3 4.3
Feasibility 3.7 1.8 2.2 2.8 2.5 2.8 2.0 4.2Teamwork/Collaboration 4.5 3.7 3.3 2.7 3.7 4.7 3.3 4.0
Totalweightedscore 3.7 2.2 3.1 2.9 2.9 2.9 2.2 4.2Table92:AveragePanelScoresforEarlyCareerSTEMProfessionalsTeamFinalProducts(InnovationOutcomes),ControlandTreatment
Control Treatment
StatisticallySignificantDifference?
Insightintochallenge 3.6 3.2 NoClarityandrelevanceofproblem 3.6 3.5 NoProblemSolvingStrategy 2.7 2.9 NoImpact 2.3 2.7 NoDevelopmentalStrategy 2.5 2.4 NoFeasibility 2.6 2.9 NoTeamwork/Collaboration 3.5 3.9 NoTotalweightedscore 3.0 3.0 No Theroleoftheexpertpanelistswastoprovideanobjectiveassessmentofthenewproducts,processesandservicesconceptuallydevelopedbytheteamsovertheir5-weekcoursesoftraining(i.e.,teaminnovationoutputs).Assuch,panelistsconsideredtheextenttowhicheachteamwasabletosurfaceinsightfulthinkingabouttheInnovationChallenge;identifyaclear,relevantandproductiveproblemtosolve;findanovelandpotentiallyimpactfulwaytosolveitschosenproblem;outlineacompellingdevelopmentalandgo-to-marketstrategyforitsinnovation;andarticulateapotentiallyfeasiblepathfromconcepttoimplementation.Inaddition,10%ofeachteam’stotalweightedscorewasbasedonthepanel’sassessmentofitsintegrationskillsthroughdistributedandcollaborativeeffort;anassessmentthatwasnecessarilyinferential,sincepanelistshadextremelylimitedopportunitytowitnessdirectevidenceofteamcollaborativeprocesses,dynamicsanddecision-making.GiventhatthepanelistswerenotinvolvedinorpresentduringthetrainingsessionsinWorcesterorSanDiegoandthattheydidnotknowwhichgroupswerecontrolortreatment,theirscoringwasagoodtestofwhethertheintegrationofarts-basedlearningintoinnovationtrainingimpactedthekindsofnew
AudienceViewpointsConsultingArtofScienceLearningResearchReport116
products,processesandservicesdevelopedbytheteams;andwhetheritimpactedthewaysteamscreativelyaddressedSTEMInnovationChallengesintheirthinkingaboutproblems,solutions,novelty,value,impact,feasibilityandotherkeyconceptsofinnovation.
TeamInnovationOutcomesFindingsSummaryItwasaveryimportantresultthattheexpertpanelistsratedthehighschoolproducts,processesandservicesofthetreatmentteamssignificantlyhigherthatthoseofthecontrolteamsintermsofinsight,clarity,problemsolvingstrategy,andpotentialimpact.Incontrast,therewerenostatisticallysignificantdifferencesbetweencontrolandtreatmentgroupsreflectedinthepanelistratingsoftheearlycareerSTEMprofessionalteams.AsmentionedearlierintheLimitationssectionofthisreport,itispossiblethatthislackoffindingsfromtheadultteamsmayresultfromusingacurriculumthatwasdevelopedspecificallyforadolescents.Furtherstudytodeterminewhetheradultfindingswouldchangewiththesubstitutionofacurriculumspecificallydesignedforusewithadultswouldbeveryuseful.Thefindingsinthissectionshowthepowerfulpotentialbenefitsofapplyingarts-basedapproachestoadolescentSTEMinnovationlearningandpractice.Theyalsounderlinethecriticalneedforfurtherstudytodeterminewhether,andinwhatways,adultfindingswouldchangewiththesubstitutionofacurriculumspecificallydesignedforusewithadults.
AudienceViewpointsConsultingArtofScienceLearningResearchReport117
References
Basadur,M.S.(1993).Impactsandoutcomesofcreativityinorganizationalsettings.InTheemergenceofadiscipline:Nurturinganddevelopingcreativity.VolumeII.(Editors:Isaksen,S.G.,Murdock,M.C.,Firestein,R.L.,&Treffinger,D.J.),Chapter12(2),278-313.NewYork:Ablex.
Basadur,M.S.,&Finkbeiner,C.T.(1985).Measuringpreferenceforideationincreativeproblem
solvingtraining.JournalofAppliedBehavioralScience,Vol.21,No.1,37-49.Basadur,M.,Graen,G.,&Wakabayashi,M.(1990).Identifyingindividualdifferencesincreative
problemsolvingstyle.TheJournalofCreativeBehavior,24(2),111-131.Epstein,R.,Schmidt,S.M.,&Warfel,R.(2008).Measuringandtrainingcreativitycompetencies:
Validationofanewtest.CreativityResearchJournal,20(1),7-12.Hackman,J.R.(2002).Leadingteams:Settingthestageforgreatperformances.Boston,MA:
HarvardBusinessSchoolPress.Heifetz,R.A.,&Linsky,M.(2002).Leadershipontheline:Stayingalivethroughthedangersof
leading.Boston,MA:HarvardBusinessSchoolPress.Kahn,K.B.(Ed.)(2013).ThePDMAhandbookofnewproductdevelopment.Hoboken,NewJersey:
JohnWiley&Sons.Retrievedfromhttp://altmannpeter.com/PDMA%20Handbook%20of%20New%20Product%20Development%20(3rd%20Edition).pdf
Katzenbach,J.R.,&Smith,D.K.(1993).Thewisdomofteams:Creatingthehigh-performance
organization.Boston,MA:HarvardBusinessSchoolPress.Norton,R.W.(1975).Measurementofambiguitytolerance.JournalofPersonalityAssessment,39,
607-619.Runco,M.A.,&Basadur,M.S.(1993).Assessingideationalandevaluativeskillsandcreativestyles
andattitudes.CreativityandInnovationManagement,2(3),166-173.Sawyer,K.(2007).Groupgenius:Thecreativepowerofcollaboration.NewYork,NY:BasicBooks.Senge,P.,&Scharmer,O.(2001).Communityactionresearch.InP.ReasonandH.Bradbury(Eds.),
HandbookofactionResearch.ThousandOaks,CA:SagePublications.Thota,H.,&Munir,Z.(2012).Keyconceptsininnovation.Boston,MA:PalgraveMacmillan.
AudienceViewpointsConsultingArtofScienceLearningResearchReport118
Appendices
AppendixA:PreWorkshopSurvey WelcometotheTransportationChallenge!Wearesopleasedtohaveaspartofthisproject.Asyouknow,thisprojectistheresearchportionofalargerNSFgrantonscienceandlearningininformalcontexts.Theresearchisanintegralportionoftheproject,andunderstandingwhoyouare,whatyouthink,andwhatyoudoisimportantinformationformakingtheprojectasuccess.Thisinformationisalsoimportanttounderstandwhetherprojectsofthistypeareworthyourtimeandtaxpayermoney.Thisparticularsurveyiscriticaltousgoingforward.Thissurveyshouldtakeapproximately20-30minutestocomplete.Thequestionsinthesurveyareaboutyou;therearenorightorwronganswers.Pleasebeassuredthatallofyourresponseswillbekeptconfidential.Noneoftheinformationyouprovidethatcouldidentifyyouindividuallywillbeincludedinthepresentationofsurveyresults.Wewouldliketoencourageyoutotakeyourtimeandanswerthesurveyquestionsbothasopenlyandhonestlyasyoucan.
We’dliketoknowyouabitbetterasyoujointhisproject.Thenextsetofquestionsarebackgroundquestionstobetterunderstandyouroutsideinterests,especiallyconnectionswithscienceandart.
1. Howoftenhaveyouparticipatedinthefollowingactivitiesinthelasttwoyears:
Activity Notatallinthelasttwoyears
Atleastonceinthelasttwoyears
Attendedalivemusic,theater,ordanceperformance Performedinorpracticedaspecificartform(e.g.,dance,singing,classicalmusic,etc.)
Attendedanartmuseumorgallery Tookaclassorlesson(whetherinoroutofschool)inanartformorartsubject
Emailed,posted,orsharedartwork(yourownorothers;includesphotos&music)
UsedTV,radio,ortheInternettoaccessartorartsprogramming
Attendedaprofessionalconferencerelatedtothearts Readanarts-focusedblog Visitedacraftsfairoravisualartsfestival Taughtanartclassorlesson Readascience-focusedblog Taughtascienceclassorlesson Attendedaprofessionalconferencerelatedtoscience
AudienceViewpointsConsultingArtofScienceLearningResearchReport119
Attendedasciencemuseum Performedascienceexperiment(informallyorformally) UsedTV,radio,ortheInternettoaccessscienceprogramming
Visitedasciencefestival Tookascienceclassorlesson(whetherinoroutofschool)
Emailed,posted,orsharedscientificinformation(e.g.,quoteorarticleofinterest,etc.)
Participatedinaneveningeventatascience-basedinstitution
2. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeordisagreewiththefollowingstatements?
a) OccasionallyIliketoworkonextremelydifficultproblemsb) Ionlyliketasksthathaveahighprobabilityofsuccessc) Idonotsharemyideaswithothersd) Ioftenreadbooksandmagazinesoutsideofmycoreinterestareae) Isometimesusemydreamsordaydreamsasasourceofnewideasf) Iamnotafraidoffailureg) Daydreamingonlywastesmytimeh) Idonotliketoworkonproblemsthathavenosolutioni) TherearespecialplaceswhereIgotothinkj) Ikeepsomethingbymybedatnight,torecordideask) Ienjoyworkingwiththesamegroupofpeopleallthetimel) Idonotneedanymorecolleaguesm) Iseektraininginnewareasn) Imakeanefforttomeetnewpeople
3. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeor
disagreewiththefollowingstatements?a) Almosteveryproblemhasasolutionb) Iliketofoolaroundwithnewideas,eveniftheyareatotalwasteoftimec) Nothinggetsaccomplishedinthisworldunlessyousticktosomebasicrulesd) Usually,themoreclearlydefinedrulesasocietyhas,thebetteroffitise) Personally,Itendtothinkthatthereisarightwayandwrongwaytodoalmosteverythingf) Idon'tneedtofinishataskbeforestartinganewtaskg) Beforeanyimportantjob,Imustknowhowlongitwilltakeh) Inaproblem-solvinggroupitisalwaysbesttosystematicallyattackaproblemi) IdonotliketogetstartedingroupprojectsunlessIfeelassuredthattheprojectwillbe
successful
AudienceViewpointsConsultingArtofScienceLearningResearchReport120
j) Inadecision-makingsituationinwhichthereisnotenoughinformationtoprocesstheproblem,Ifeelveryuncomfortable
4. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeordisagreewiththefollowingstatements?
a) Thereistoomuchwastedtimeinthecreativeprocessb) Brilliantideascomefromsingle‘AHA!’moments,notfromworkingthroughthecreative
processc) Emotionsdetractfromeffectivecommunicationd) Itisalwayscounterproductivewhenconflictarisesduringcollaborativeworke) Agoodcommunicatorcancommunicateeffectivelythesamewayineverysituationf) Collaborationisrarelyworththetimeittakesg) Noteveryoneiscapableofcreativityh) Thelesslifeexperienceyouhavethelessyouhavetoofferwhencollaboratingwithothersi) Peoplearemorelikelytoproduceeffectivesolutionsthroughcompetition,ratherthan
throughcollaborationj) Individualsaremorelikelythangroupstocomeupwithtrulyoriginalideask) Alotofpeoplewhothinktheyareeffectivecommunicatorsjusttalkalotl) Tobecreativeyoumustbeartisticm) Creativityissomethingdonebyindividuals,notsomethingthathappensonagroupleveln) Communicationispredominantlyverbalo) Peoplewhoareoutgoingarenaturallybettercommunicators
5. Onascalefrom1(Stronglydisagree)to5(Stronglyagree,pleaseratethefollowingsentences:a) Ishoulddosomeprejudgmentofmyideasbeforetellingthemtoothers.b) Weshouldcutoffideaswhentheygetridiculousandgetonwithit.c) Ifeelthatpeopleatworkoughttobeencouragedtosharealltheirideas,becauseyounever
knowwhenacrazy-soundingonemightturnouttobethebest.d) Onenewideaisworthtenoldones.e) Qualityisalotmoreimportantthanquantityingeneratingideas.f) Agroupmustbefocusedandontracktoproduceworthwhileideasg) Lotsoftimecanbewastedonwildideas.h) Ithinkeveryoneshouldsaywhateverpopsintotheirheadwheneverpossible.i) Iliketolistentootherpeople’scrazyideassinceeventhewackiestoftenleadstothebest
solution.j) Judgmentisnecessaryduringideagenerationtoensurethatonlyqualityideasare
developedk) Youneedtobeabletorecognizeandeliminatewildideasduringideageneration.l) Ifeelthatallideasshouldbegivenequaltimeandlistenedtowithanopenmindregardless
ofhowzanytheyseemtobe.
AudienceViewpointsConsultingArtofScienceLearningResearchReport121
m) Thebestwaytogeneratenewideasistolistentoothersthentailgateoraddon.n) Iwishpeoplewouldthinkaboutwhetherornotanideaispracticalbeforetheyopentheir
mouth.
6. Onemoresetofquestionsabouthowyouliketotackleproblems.Onascalefrom1(Stronglydisagree)to7(Stronglyagree)onthesequestionsonyourpersonality:
a) Iliketogetthingsstartedbygettinginvolved,gatheringinformation,questioning.b) Ilikeimaginingthepossibilitiesandsensingallkindsofnewproblemsandopportunities.c) Icanseegoodandbadsidestoalmostanyfact,ideaorissue.d) Iamcomfortablewithsituationswerenoteverythingisclear.e) I’mwillingtoletotherstakecareofthedetails.f) Itendtoformquickassociations,defineproblemsandconceptualizenewideas,
opportunitiesandbenefits.g) Iamgoodatininductivereasoning,inpullingtogetherseeminglyunrelatedobservations
intoanintegratedsolution.h) Idon’tlikegoingforwarduntilIhaveasoundunderstandingofthesituation.i) Iwouldprefernottohavetoprioritizeamonggoodornotfullyunderstoodalternatives.j) Ipreferideasratherthanmovingtoaction.k) Idobestinsituationswherethereisasinglecorrectanswerorbestsolutiontoaproblem.l) Icansortthroughlargeamountsofdataandpinpoint“what’swrong”inagivensituation.m) Iamconfidentofmyabilitytomakeasoundevaluationandselectthebestsolutiontoa
problemn) Itendtolackpatiencewithambiguity.
Iprefernotspendingtoomuchtimethinkingaboutotherideasandpointsofview,orhowdifferentproblemsrelatetooneanother.
o) Ilikebecominginvolvedinnewexperiences.p) Iliketotrythingsoutratherthan“mentallytest”them.q) Iconsidermyselfarisk-taker:Idon’tneedtounderstandsomethingcompletelybeforeIact.r) I’mwillingtotryasmanydifferentapproachesasnecessaryuntilIfindonethatis
sufficientlyacceptabletothoseaffectedbytheproblem.s) Itendtobeenthusiastic,butcanbeimpatientasItrytoactonplans.
7. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouarean
innovatorinyourschoolwork?
8. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouareaninnovatorinyourprofessionallife?
9. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouarean
innovatorinyourpersonallife,includinghobbiesandinterests?
AudienceViewpointsConsultingArtofScienceLearningResearchReport122
10. Pleaseexplainyourratingsonwhetheryouseeyourselfasaninnovatorornot.
11. Workingwithotherclasses,we’velearnedthatindividualshaveverydifferentandpersonal
definitionsofinnovation.Whatisyourdefinitionofinnovation?Youarealmostdone!Thefinalsetofquestionsaredemographicinnaturethatwecollecttoknowthatadiversesampleofpeopleparticipatedwithinthisproject.Again,yourresponseswillremainanonymous.12. Areyou...
a) Maleb) Femalec) Other/Prefernottosay.
13. Whichethniccategorydoyoumostidentifywith?(PleasecheckALLthatapply)Thisinformationis
important,aspartoffederalgrant,weneedtoshowwearereachingabalancedaudience.
a) AfricanAmericanorBlackb) AmericanIndianorAlaskaNativec) Asiand) CaucasianorWhitee) HispanicorLatinof) NativeHawaiianorotherPacificIslanderg) Other:(pleasespecify)h) Prefernottoanswer
14. Whatisthehighestlevelofeducationthatyou'vecompleted?(PleasecheckONE)
a) Lessthanhighschool(I'mstillenrolledinhighschool)b) Lessthanhighschool(I'mnolongerenrolledinhighschool)c) HighSchool/GEDd) Communitycollege/technicaltrainingorcertificatee) Collegedegree(BA/BS)f) GraduateorPostgraduatedegree
15. Thankyouverymuchforcompletingthispre-survey.Wearelookingforwardtoyourparticipation
inthechallengeoverthecomingweeks!Ifthereisanythingelseyouwouldliketosharewithusatthistimepleasedosobelow.
AudienceViewpointsConsultingArtofScienceLearningResearchReport123
AppendixB:PostWorkshopSurvey Wehavebeensopleasedtohaveyouparticipateinthischallenge!Nowforthefinalreflection.We’dliketoaskyousomequestionsaboutyou,yourteam,andyourthoughtsaboutthechallenge.Asyouknow,thisprojectistheresearchportionofalargerNSFgrantonscienceandlearningininformalcontexts.Theresearchisanintegralportionoftheproject,andunderstandingwhoyouare,whatyouthink,whatyoucandoisimportantinformationformakingtheprojectasuccess.Thisinformationisalsoimportanttounderstandwhetherprojectsofthistype(research)areworthyourtimeandtaxpayermoney.Thisparticularsurveyiscriticaltotheresearch.Thissurveyshouldtakeapproximately20-30minutestocomplete.Thequestionsinthesurveyareaboutyou;therearenorightorwronganswers.Pleasebeassuredthatallofyourresponseswillbekeptanonymous.Noneoftheinformationyouprovidethatcouldidentifyyouindividuallywillbeincludedinthepresentationofsurveyresults.Wewouldliketoencourageyoutotakeyourtimeandanswerthesurveyquestionsbothasopenlyandhonestlyasyoucan.1. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeor
disagreewiththefollowingstatements?a) OccasionallyIliketoworkonextremelydifficultproblemsb) Ionlyliketasksthathaveahighprobabilityofsuccessc) Idonotsharemyideaswithothersd) Ioftenreadbooksandmagazinesoutsideofmycoreinterestareae) Isometimesusemydreamsordaydreamsasasourceofnewideasf) Iamnotafraidoffailureg) Daydreamingonlywastesmytimeh) Idonotliketoworkonproblemsthathavenosolutioni) TherearespecialplaceswhereIgotothinkj) Ikeepsomethingbymybedatnight,torecordideask) Ienjoyworkingwiththesamegroupofpeopleallthetimel) Idonotneedanymorecolleaguesm) Iseektraininginnewareasn) Imakeanefforttomeetnewpeople
2. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeor
disagreewiththefollowingstatements?a) Almosteveryproblemhasasolutionb) Iliketofoolaroundwithnewideas,eveniftheyareatotalwasteoftimec) Nothinggetsaccomplishedinthisworldunlessyousticktosomebasicrulesd) Usually,themoreclearlydefinedrulesasocietyhas,thebetteroffitise) Personally,Itendtothinkthatthereisarightwayandwrongwaytodoalmosteverythingf) Idon'tneedtofinishataskbeforestartinganewtaskg) Beforeanyimportantjob,Imustknowhowlongitwilltake
AudienceViewpointsConsultingArtofScienceLearningResearchReport124
h) Inaproblem-solvinggroupitisalwaysbesttosystematicallyattackaproblemi) IdonotliketogetstartedingroupprojectsunlessIfeelassuredthattheprojectwillbe
successfulj) Inadecision-makingsituationinwhichthereisnotenoughinformationtoprocessthe
problem,Ifeelveryuncomfortable
3. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyouagreeordisagreewiththefollowingstatements?
a) Thereistoomuchwastedtimeinthecreativeprocessb) Brilliantideascomefromsingle‘AHA!’moments,notfromworkingthroughthecreative
processc) Emotionsdetractfromeffectivecommunicationd) Itisalwayscounterproductivewhenconflictarisesduringcollaborativeworke) Agoodcommunicatorcancommunicateeffectivelythesamewayineverysituationf) Collaborationisrarelyworththetimeittakesg) Noteveryoneiscapableofcreativityh) Thelesslifeexperienceyouhavethelessyouhavetoofferwhencollaboratingwithothersi) Peoplearemorelikelytoproduceeffectivesolutionsthroughcompetition,ratherthan
throughcollaborationj) Individualsaremorelikelythangroupstocomeupwithtrulyoriginalideask) Alotofpeoplewhothinktheyareeffectivecommunicatorsjusttalkalotl) Tobecreativeyoumustbeartisticm) Creativityissomethingdonebyindividuals,notsomethingthathappensonagroupleveln) Communicationispredominantlyverbalo) Peoplewhoareoutgoingarenaturallybettercommunicators
4. Onascalefrom1(Stronglydisagree)to5(Stronglyagree,pleaseratethefollowingsentences:a) Ishoulddosomeprejudgmentofmyideasbeforetellingthemtoothers.b) Weshouldcutoffideaswhentheygetridiculousandgetonwithit.c) Ifeelthatpeopleatworkoughttobeencouragedtosharealltheirideas,becauseyounever
knowwhenacrazy-soundingonemightturnouttobethebest.d) Onenewideaisworthtenoldones.e) Qualityisalotmoreimportantthanquantityingeneratingideas.f) Agroupmustbefocusedandontracktoproduceworthwhileideasg) Lotsoftimecanbewastedonwildideas.h) Ithinkeveryoneshouldsaywhateverpopsintotheirheadwheneverpossible.i) Iliketolistentootherpeople’scrazyideassinceeventhewackiestoftenleadstothebest
solution.j) Judgmentisnecessaryduringideagenerationtoensurethatonlyqualityideasare
developed
AudienceViewpointsConsultingArtofScienceLearningResearchReport125
k) Youneedtobeabletorecognizeandeliminatewildideasduringideageneration.l) Ifeelthatallideasshouldbegivenequaltimeandlistenedtowithanopenmindregardless
ofhowzanytheyseemtobe.m) Thebestwaytogeneratenewideasistolistentoothersthentailgateoraddon.n) Iwishpeoplewouldthinkaboutwhetherornotanideaispracticalbeforetheyopentheir
mouth.
5. Onemoresetofquestionsabouthowyouliketotackleproblems.Onascalefrom1(Stronglydisagree)to7(Stronglyagree)onthesequestionsonyourpersonality:
a) Iliketogetthingsstartedbygettinginvolved,gatheringinformation,questioning.b) Ilikeimaginingthepossibilitiesandsensingallkindsofnewproblemsandopportunities.c) Icanseegoodandbadsidestoalmostanyfact,ideaorissue.d) Iamcomfortablewithambiguity.e) I’mwillingtoletotherstakecareofthedetails.f) Itendtoformquickassociations,defineproblemsandconceptualizenewideas,
opportunitiesandbenefits.g) Iexcelininductivereasoning,indistillingseeminglyunrelatedobservationsintoan
integratedsolution.h) Idon’tlikeproceedinguntilIhaveasoundunderstandingofthesituation.i) Iwouldprefernottohavetoprioritizeamonggoodornotfullyunderstoodalternatives.j) Ipreferideasratherthanmovingtoaction.k) Idobestinsituationswherethereisasinglecorrectansweroroptimalsolutiontoa
problem.l) Icansortthroughlargeamountsofdataandpinpoint“what’swrong”inagivensituation.m) Iamconfidentofmyabilitytomakeasoundevaluationandselectthebestsolutiontoa
problemn) Itendtolackpatiencewithambiguity.
Iprefernotspendingtoomuchtimethinkingaboutotherideasandpointsofview,orhowdifferentproblemsrelatetooneanother.
o) Ilikebecominginvolvedinnewexperiences.p) Iliketotrythingsoutratherthan“mentallytest”them.q) Iconsidermyselfarisk-taker:Idon’tneedtounderstandsomethingcompletelybeforeIact.r) I’mwillingtotryasmanydifferentapproachesasnecessaryuntilIfindonethatis
sufficientlyacceptabletothoseaffectedbytheproblem.s) Itendtobeenthusiastic,butcanbeimpatientasItrytoactonplans.
6. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouarean
innovatorinyourschoolwork?
7. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouareaninnovatorinyourprofessionallife?
AudienceViewpointsConsultingArtofScienceLearningResearchReport126
8. Onascalefrom1(Stronglydisagree)to7(Stronglyagree),towhatextentdoyoufeelyouarean
innovatorinyourpersonallife,includinghobbiesandinterests?
9. Pleaseexplainyourratingsonwhetheryouseeyourselfasaninnovatorornot.
10. Hasyourperceptionofyourselfasaninnovatorchangedduringyourparticipationinthisproject?Inwhatways?
11. Onascalefrom1(StrongNegativeimpact)to7(StrongPositiveImpact),towhatextentdidparticipatingintheTransportationChallengechange:
a) youroverallabilitytothinkcreatively?b) Whetherandhowyoudocumentideasastheypopup?c) thewayyoubrainstorm?d) originalityofthinking?e) howyoudescribeandelaborateonyourthinkingtoothers?f) howyouworkwithawiderangeofindividuals?g) yourcomfortinplacingyourselfintodifficultorunfamiliarsituations?h) yourcomfortinambiguoustasks?i) whetheryouthinkofyourselfascreative?j) howmanystrategiesyoucomeupwithwhenproblem-solving?
12. Nowthattheprojectisover,wewouldlikeyoutoratethecontributionofeachmemberofyour
grouptothecompletionofthetasksandoverallproject.Yourratingswillbeanonymousandwillnotbesharedwithothermembersofthegroupatanytime.Theywillonlybeusedforanalysispurposes,andcombinedwithotherresponses.
Overall,duringthisproject…
CONTRIBUTEDVERYLITTLE
1
2
3
4
5
6
CONTRIBUTEDALOT7
Member#1,Yourself: ! ! ! ! ! ! !Member#2,Name: ! ! ! ! ! ! !Member#3,Name: ! ! ! ! ! ! !Member#4,Name: ! ! ! ! ! ! !Member#5,Name: ! ! ! ! ! ! !Member#6,Name: ! ! ! ! ! ! !Member#7,Name: ! ! ! ! ! ! !Member#8,Name: ! ! ! ! ! ! !
Thankyouverymuchforcompletingoursurveyandthisproject!Wereallyappreciateyourtimeandthoughtfulness.Doyouhaveanyfurthercomments,thoughtsorquestionsforus?
AudienceViewpointsConsultingArtofScienceLearningResearchReport127
AppendixC:ObservationRating TEAM:___Date(dd/mm/yy):_______ObserverInitials:_______________EachObservationSheetrepresentsonegroupobservedideallyfor20-30minutesduringanon-taskgrouptime.Onceobservationbegins,itdoesn’tstopeveniftheygroupveersofftask.GROUPACTIVITY#:________StarttimeofObservation:______EndTimeofObservation:_______
#1:
#2:
#3:
#4:
#5:
#6:
#7:
#8:
#9:
#10:
#11:
ParticipantContributiontotheTaskRating(1-7) __ __ __ __ __ __ __ __ __ __ __
• Sharesleadershipaccordingtoknowledge/skill
! ! ! ! ! ! ! ! ! ! !
• Active,engagedfollower ! ! ! ! ! ! ! ! ! ! !
• Emotionallyintelligentbehaviors,interpersonalrelationshipsanddynamics
! ! ! ! ! ! ! ! ! ! !
• Empathiclistening(Opennesstootherpointsofview,acknowledgesothersviews)
! ! ! ! ! ! ! ! ! ! !
• Mutualrespect ! ! ! ! ! ! ! ! ! ! !• Trustinmovingtowardsasolution ! ! ! ! ! ! ! ! ! ! !
• Transparentincommunicationsandinformation
! ! ! ! ! ! ! ! ! ! !
• Abilitytodisagreeanddisputeproductively ! ! ! ! ! ! ! ! ! ! !
• Definingacommonpurpose(particularlyatthesolution,buy-inasanindicator)
! ! ! ! ! ! ! ! ! ! !
• Creatingacultureofmutualaccountability ! ! ! ! ! ! ! ! ! ! !
• Productivelymanagesdisruptionwithinthegroup,reframingcontextordirection
! ! ! ! ! ! ! ! ! ! !
AudienceViewpointsConsultingArtofScienceLearningResearchReport128
AppendixD:CollaborationRating Name:_______________Date(dd/mm/yy):_________GroupNumber:__________Pleaseanswerthequestionsbelowaboutyourgroupashonestlyaspossible.Theothergroupmemberswillnotseeyourresponsesandtheywillallbecombinedforanalysis.
Duringtoday’stask(s),ourgroup…
STRONGLYDISAGREE
1
2
3
NEITHERAGREENORDISAGREE
4
5
6
STRONGLYAGREE7
• Sharedleadershipbetweenpeople,basedonknowledgeandskill ! ! ! ! ! ! !
• Didagoodjobfollowingtheintentofthegroup,ratherthaneachperson’sindividualthinking
! ! ! ! ! ! !
• Wasawareofandtookintoaccounthoweachgroupmemberwasfeeling
! ! ! ! ! ! !
• Listenedtoandacknowledgedeveryone’spointsofview ! ! ! ! ! ! !
• Showedmutualrespecttoeachother ! ! ! ! ! ! !
• Trustedthatwewouldeventuallycomeupwithagoodsolution ! ! ! ! ! ! !
• Wastransparentwitheachotherwhenwecommunicatedandsharedinformation
! ! ! ! ! ! !
• Wasabletodisagreeanddisputeproductively ! ! ! ! ! ! !
• Definedthecommonpurposewewereworkingtowards ! ! ! ! ! ! !
• Createdacultureofmutualaccountability,wherewewereallresponsibleforcompletingthetask
! ! ! ! ! ! !
• Productivelymanagedanydisruptionwithinthegroup(ifapplicable)
! ! ! ! ! ! !
• Wasabletosuccessfullycompletethetask(s)assignedtoustoday. ! ! ! ! ! ! !
AudienceViewpointsConsultingArtofScienceLearningResearchReport129
AppendixE:TransferabilitySurvey It’sbeenseveralmonthssinceyourparticipationintheWorcesterInnovationChallenge.Nowthatsometimehaspassed,we’dliketoaskyousomequestionsaboutyourparticipation.1. Towhatextenthaveyoubeenthinkingaboutwhatyoulearnedaboutinnovationandcreativity
duringtheWorcesterInnovationChallengePleasecheckoneonly.
Haven’tthoughtaboutitatall
Thoughtaboutitonceortwice
Havethoughtaboutitweekly
Havethoughtaboutitseveraltimesaweek
Havethoughtaboutitdaily
2. WhatwereyourinitialreasonsfordecidingtoparticipateintheChallenge.Whatmotivatedyoutodoit?
3. DidparticipatingintheChallengehaveanyeffectonhowyouhavebeenthinkingaboutsolutions
forothertopicsorissues?
1–Notatall
2 34–Tosome
extent5 6
7-Agreatdeal
a. Canyougiveanexampleofhowithaschanged?
4. Doyoufeelinthisworkshopyouhaddoactivitiesthatwereoutofyourcomfortzone?
1–Notatall
2 34–Tosome
extent5 6
7-Agreatdeal
a. Whydidyougivetheratingyoudid?
5. TowhatextenthaveyoubeenabletoapplytheInnovationChallengeexperiencetoyourcurrentworkorvolunteeractivities?
1–Notatall
2 34–Tosomeextent
5 67-Agreat
deal
NotApplicable
6. Whichpartsofthechallengehaveyoubeenabletouseinyourcurrentworkorvolunteer
activities?Pleasegiveasmanyexamplesasyoucan.
AudienceViewpointsConsultingArtofScienceLearningResearchReport130
7. TowhatextentdoyouthinkyouwillbeabletoapplyyourInnovationChallengeexperiencetoyourfutureworkorvolunteeractivities?
1–Notatall
2 34–Tosomeextent
5 67-Agreat
deal
NotApplicable
a. Ifyes,inwhatways?
8. TowhatextenthaveyoubeenabletoapplytheInnovationChallengeexperiencetoyourcurrentschoolorextracurricularsituation?
1–Notatall
2 34–Tosomeextent
5 67-Agreat
deal
NotApplicable
9. Whichpartsofthechallengehaveyoubeenabletouseinyourcurrentschoolorextracurricular
activities?Pleasegiveasmanyexamplesasyoucan.
10. TowhatextentdoyouthinkyouwillbeabletoapplyyourInnovationChallengeexperienceto
yourfutureschoolorextracurricularsituation?
1–Notatall
2 34–Tosomeextent
5 67-Agreat
deal
NotApplicable
a. Ifyes,inwhatways?
11. TowhatextenthaveyoubeenabletoapplyyourInnovationChallengeexperiencetoyourcurrent
home/personallife?
1–Notatall
2 34–Tosome
extent5 6
7-Agreatdeal
a. Inwhatways?
12. Ifyoufeltyoubenefitedfromthischallenge,wereparticularattributesaboutyourbackgroundor
situationthatmadeitusefultoyou?We’reinterestedinwhetherthosethatbenefitfromthistraininghaveparticularcharacteristicsorbackgroundincommon.
AudienceViewpointsConsultingArtofScienceLearningResearchReport131
13. WhatwastheonethingyouexperiencedintheChallengethatyouthinkwillbemosthelpfultoyouinthefuture,andwhy?
AudienceViewpointsConsultingArtofScienceLearningResearchReport132
AppendixF:PreWorkshopCreativitySkillsTaskWorcester YourName:__________________________________Group: ☐ Morning �AfternoonWelcometotheWorcesterChallenge!We’resopleasedtohaveyouhere.Wehaveabriefwarm-uptaskthatweneedyoutocomplete.YourWarm-UpChallengeisasfollows:Worcester(substituteyourtownifyou’renotlivinginWorcester)needstransportationalternativestoenhanceitseconomicproductivity,connectitsneighborhoodsandcommunities,andimprovethequalityoflifeforitsresidents.Duringthenext15minutes,dothefollowing:A) Listupto5specificproblemsrelatedtothischallengethatyou'dliketofix.
1.2.3.4.5.
B) Fromthatlistabove,selecttheoneproblemyouwouldmostliketofix,andexplainwhy
youchosethatparticularproblemfromthelistyoucreated.Itmayhelptofocusontheparticularcharacteristic(s)oftheproblemthatmakestheproblemyouselectedeitherparticularlyimportantorparticularlyfixable.
AudienceViewpointsConsultingArtofScienceLearningResearchReport133
C) Listupto5possiblesolutionstotheproblemyou'vechoseninanswer‘B’above.
1.2.3.4.5.
D) Fromthelistof5possiblesolutions,selecttheonesolutionyouwouldmostliketodevelop.Explainwhyyouchosethissolution,focusingontheparticularcharacteristic(s)ofthesolutionthatseemstohavemadeitmostappropriateforaddressingtheproblem,givingasmuchevidenceaspossibleforyourdecision.
AudienceViewpointsConsultingArtofScienceLearningResearchReport134
AppendixG:PreWorkshopCreativitySkillsTaskSanDiego YourName:__________________________________
Group: �Morning �Afternoon
WelcometotheSanDiegoChallenge!We’resopleasedtohaveyouhere.Wehaveabriefwarm-upactivitythatwe’dlikeyoutocomplete.
SanDiego’sInnovationChallengeisasfollows:
SanDiegoneedsinnovationsthatwillreducethegapbetweenitsregionalwatersupplyandthedemandsofitsindustrial,agriculturalandresidentialusers.
Duringthenext7minutes,please:
A) Listupto5specificproblemsrelatedtothischallengethatyou'dliketofix.
1.
2.
3.
4.
5.
B) Fromthatlist,selecttheoneproblemyouwouldmostliketofixandbrieflyexplainwhyyouchosethatparticularproblem.Itmayhelptofocusoncharacteristicsoftheproblemyouselectedthatmakeiteitherparticularlyimportantorparticularlyfixable.
AudienceViewpointsConsultingArtofScienceLearningResearchReport135
InnovationChallenge
January10,2015
YourName:__________________________________
Group: � Morning � Afternoon
Duringthenext7minutes,please:
C) Listupto5possiblesolutionstotheproblemyou'vechosenasanswer“B”onthepreviouspage.
1.
2.
3.
4.
5.
D) Fromthislist,pleaseselecttheonesolutionyouwouldmostliketodevelopandbrieflyExplainwhyyouchosethissolution.Pleasefocusonthecharacteristicsofthesolutionyouchosethatmotivatedyoutochooseit,givingasmuchevidenceaspossibletosupportyourdecision.
AudienceViewpointsConsultingArtofScienceLearningResearchReport136
AppendixH:PostWorkshopCreativitySkillsTask Wehavesoenjoyedtheseconversationsoverthelastfewweeks.Asoneoftheclosingactivities,we’dlikeyoutoworkonanotherquicktask.YourWrap-UpChallengeisasfollows:ImagineatownsimilarsizeandcompositionofWorcester/SanDiego,wheretwentypercentofthepopulationisaffectedbyfoodinsecurity,lacksfairandequitableaccesstoasufficientquantityofaffordable,nutritiousfoodforallcitizens.Duringthenext7minutes,dothefollowing:A) Listupto5specificproblemsrelatedtothischallengethatyou'dliketofix.
1.2.3.4.5.
B) Fromthatlistabove,selecttheoneproblemyouwouldmostliketofix,andexplainwhy
youchosethatparticularproblemfromthelistyoucreated.Itmayhelptofocusontheparticularcharacteristic(s)oftheproblemthatmakestheproblemyouselectedeitherparticularlyimportantorparticularlyfixable.
AudienceViewpointsConsultingArtofScienceLearningResearchReport137
Duringthenext7minutes,dothefollowing:C) Listupto5possiblesolutionstotheproblemyou'vechoseninanswer‘B’above.
1.2.3.4.5.
D) Fromthelistof5possiblesolutions,selecttheonesolutionyouwouldmostliketo
develop.Explainwhyyouchosethissolution,focusingontheparticularcharacteristic(s)ofthesolutionthatseemstohavemadeitmostappropriateforaddressingtheproblem,givingasmuchevidenceaspossibleforyourdecision.
AudienceViewpointsConsultingArtofScienceLearningResearchReport138
AppendixI:CreativitySkillsTaskRubric SectionA:Listupto5problemsEachproblemwillbescoredeither0or1,asperthescoringtablebelow.Atotal,from0toamaximumof5points,willbeassignedforeachrespondent.SCORE1:SectionAScoringTable
0points 1point• Restatementofanotherproblem
onthelist• Answernotfocusedonthetopic• Listedonlysolutions• Commentsabouthowtogoaboutidentifyingproblemsorsolutions
• Commentssuggestingtheparticipantsdidnotreadorunderstandtheinstructions
1pointperdistinctproblemmeetingthefollowingfivecriteria(nopartialcreditperproblem).
• Theintentoftheproblemstatementisreasonablyclear• Theproblemstatementisnotjustarestatementofthechallenge• Aproblemisstated,ratherthanasolution• Theproblemisdistinctfromothersidentifiedbythesameperson• Theproblemrelatestothechallenge
Example:CarpollutionandfumesDamagedroadsSlowconstructionBikeandcaraccidentsSlowmovingtrafficMaximumpossiblescoreof5.
SCORE2:IdeaClusterTheIdeaClusterisacategoricalcode,withmultiplecategoriespossible.Foroptimumcodingpurposes,therewillbenomorethan10categories.Thiscodeisalevelhigherthanthedistinctsolution,sothatmultipledistinctproblemsmayfitwithinasingleideacluster.Forexample,ideaclusterswithinTransportationproblemsetsmayinclude,butarenotlimited:
• Safety• Poorenvironmentalconditions(suchaspollution)• Poorqualitypublicspaces• Availabilityofpublictransportation• Affordabilityofpublictransportation
SectionB:ExplainwhyyouchosetheproblemSCORE3:StrengthofProblemStatement0-2scoringtokeeppeopleinthegameasmuchaspossible.Inordertokeeppeopleinthegameasmuchaspossible,aproblemnotpreviouslylistedinSectionAisgivenequalweighttoonethatwaslistedinSectionA.Further,ascoreof1isgiventoanalignedproblemevenifitsexpressionisinferentialinnature,solongasitisclear,specificanddistinct.Iftheindividualreceivesaboveascoreofzero,moveontoscore4.
• 2=aclear,distinct,specific,explicitlywordedproblemalignedtothechallenge.• 1=aclear,distinct,specificproblemalignedtothechallengebutinferentialinexpression.• 0=aresponsethatfailedtomeetthecriteriafor1or2points.
SCORE4:Acountofeachvalid/logically-connectedreasongivenforchoosingtheproblem-onepointperreason.Thereisnolimittothepossiblescore.
AudienceViewpointsConsultingArtofScienceLearningResearchReport139
SCORE5:Acountofthedifferentrationalesfortheproblemchoiceoffered(categoricalandmultiplecategoriespossible):impact,feasibility,personalengagement,originality/uniqueness,ancillarybenefits.Theseshouldalsobecodedbycategory.Note:ananswermighthavemultiplereasonscodedasthesamecategory,andshouldonlyreceiveonepointperdifferentrationalecategory.SectionC:Listupto5solutions(seenotebelowaboutmatchbetweenSectionCandSectionsA/BEachsolutionwillbescored0or1,asperthescoringtablebelow.SCORE6a:SectionCScoringTable
0points 1point• Restatementofchallenge• Statesaproblemratherthan
asolution• Restatesanothersolutionon
thelist• Answervagueornotfocused
onthetopic• Commentsabouthowtogo
aboutidentifyingproblemsorsolutions
• Commentssuggestingtheparticipantsdidnotreadorunderstandtheinstructions
1pointperdistinctsolutionmeetingthefollowingfourcriteria(nopartialcreditpersolution).
• Theintentofthesolutionstatementisreasonablyclear• Thesolutionstatementgoesbeyondthepurelyaspirational• Thesolutionstatementclearlyrepresentsa“how”responsetothe
problemidentifiedinB• Thesolutionisdistinctfromothersidentifiedbythesameperson
1pointperdistinctsolution(nolimit).Example:IncreasepublicbusscheduleDecreasepriceofbusticketIncreasenumberofbuses
SCORE6b
0points 1point• Restatementofchallenge• Statesaproblemratherthan
asolution• Restatesanothersolutionon
thelist• Answervagueornotfocused
onthetopic• Commentsabouthowtogo
aboutidentifyingproblemsorsolutions
• Commentssuggestingtheparticipantsdidnotreadorunderstandtheinstructions
1pointperdistinctsolutionmeetingthefollowingfourcriteria(nopartialcreditpersolution).
• Theintentofthesolutionstatementisreasonablyclear• Thesolutionstatementgoesbeyondthepurelyaspirational• Thesolutionstatementclearlyrepresentsa“how”responsetosome
challenge-relatedproblemotherthantheproblemidentifiedinSectionB.
• Thesolutionisdistinctfromothersidentifiedbythesameperson1pointperdistinctsolution(nolimit).Example:IncreasepublicbusscheduleDecreasepriceofbusticketIncreasenumberofbuses
SCORE6c–totalof6aand6bSectionD:ExplainwhyyouchosethesolutionScore7:SolutionStatementStrength0-2scoringtokeeppeopleinthegameasmuchaspossible.Inordertokeeppeopleinthegameasmuchaspossible,asolutionnotpreviouslylistedinSectionCisgivenequalweighttoonethatwaslisted
AudienceViewpointsConsultingArtofScienceLearningResearchReport140
inSectionC.Further,ascoreof1isgiventoansolutionalignedwithadifferentproblem,solongasthatproblemisalignedwiththechallenge.Iftheindividualreceivesaboveascoreofzero,moveontoscore8.
• 2=aclear,distinct,specificpossiblesolutionthataddressesthechallengebysolvingtheproblempreviouslyidentifiedinSectionB.
• 1=aclear,distinct,specificpossiblesolutionthataddressesthechallengebysolvingsomeotherchallenge-relatedproblem
• 0=aresponsethatfailedtomeetthecriteriafor1or2pointsSCORE8:Countofthenumberofreasonsgiven,includingnumberof“evidence”statementswhentheyaretherewithoutreasonsSCORE9:TypeofRationaleofferedforSolutionChoice(categoricalandmultiplecategoriespossible):impact,feasibility,personalengagement,originality/uniqueness,comparativeanalysis,ancillarybenefits.SCORES10&11:SpecificityofSolution
• What(specifically)doestheparticipantproposetodo.(assign0pointsforunclear/incoherentproposal,1pointforclearbutwithoutdetail,2pointsforadditionallevelofdetail).
• How(specifically)doestheparticipantproposetodoit.(assign0pointsofthequestionisnotaddressed,1pointforageneric/non-specificwayofaddressingit,2pointsforadditionalspecificity).
SCORE12:IdeaClusteringTheIdeaClusterisacategoricalcode,withmultiplecategoriespossible.Thiscodeisalevelhigherthanthedistinctsolution,sothatmultipledistinctsolutionsmayfitwithinasingleideacluster.Forexample,ideaclusterswithinTransportationproblemsetsmayinclude,butarenotlimited:
• Safety• Improvedenvironmentconditions(suchaspollution)• Improvedpublicspaces• Availabilityofpublictransportation• Affordabilityofpublictransportation
Additionaldetails:
• StudentsareeligibletoreceivecreditforananswertoBorDeveniftheydidnotcompleteAorCiftheystatedorimpliedaproblem/solution.
AudienceViewpointsConsultingArtofScienceLearningResearchReport141
AppendixJ:BusinessCase
Teamnumber:_____________PROBLEMSTATEMENT:SOLUTIONTITLE:1. SOLUTIONOVERVIEW–Describethespecificsolutionyouwanttodevelop.
2. RELEVANCE–Howistheproposedsolutionrelevanttothe[Worcestertransportationchallenge/San
Diegowaterchallenge]?
3. NEWNESS–Describespecificallyhowthesolutionisneworinnovative.
4. MARKET
a. Whoisthecustomer?
b. Whatisthevalueproposition?Whywillcustomersbuythisinnovation?
c. Howlargeisthepotentialmarket?
AudienceViewpointsConsultingArtofScienceLearningResearchReport142
d. Howmanycustomerswillhavepurchasedthisafteroneyearonthemarket?
e. Howmuchwillyourcustomersbewillingtopayforyourinnovation?
f. Whatisthebasisforyourestimatestoc,dandeabove?
5. IMPACT–What’sthecaseforyourinnovationhavingasignificantdirectimpactonthechallenge?
6. FEASIBILITY
a. Whatkeystepsareneededtodevelopyoursolutionfromconcepttomarket?Howlongwouldtheytake?
b. Arethereobstaclesthatcallintoseriousquestionthefeasibilityoftheproposedsolution?
Howmightyouaddressthem?
c. Whatskillswillbeneededforthedevelopmentteam?
d. Aretherepotentialpartnerswhocouldhelp?
e. Howwillyourinnovationgeneraterevenuestosustainitselfinthemarket?
AudienceViewpointsConsultingArtofScienceLearningResearchReport143
AppendixK:OutputScoringSheet Panelist/Reviewername:_________________GroupnumberID:_____________1.INSIGHTINTOCHALLENGE-Graspoftransportationneeds,conditionsandopportunities.[Overallweight=15%]
5 Outstanding(Conceptshowsexceptionalinsightintoactualconditionsandopportunities,andmeetssignificantunmetneeds)
4
3 Acceptable(Conceptaddressesthechallengebysolvingrelevantunmetproblems/needs)
2
1 Limited(Conceptdeficientinitslackofgroundinginactualconditionsorfailuretoaddressrelevantunmetneeds)
InitialRating:________
Ratingafterdiscussion:______________
Comments:2-CLARITYANDRELEVANCEOFPROBLEM[Overallweight=15%]
5 Outstanding(Problemisveryclearlydefinedandrelevanttothechallenge)
4
3 Acceptable(Problemdefinitionhasreasonableclarityandrelevance)
2
1 Limited(Problemasstatedisunclearorirrelevanttothechallenge)
InitialRating:__________
Ratingafterdiscussion:______________
Comments:3-PROBLEMSOLVINGSTRATEGY[Overallweight=25%]
5 Outstanding(Solutionisahighlyoriginalandpotentiallyfruitfulwayofaddressingtheproblem)
4
3 Acceptable(Solutionhaselementsofnoveltyandseemslikelytopartiallyaddresstheproblem)
2
AudienceViewpointsConsultingArtofScienceLearningResearchReport144
1 Limited(Solutionlacksnoveltyorisunlikelytomeaningfullyaddresstheproblem)
InitialRating:_________
Ratingafterdiscussion:______________
Comments:4-IMPACT:assessedbyvalueproposition,potentialmarketsizeandpenetration.[Overallweight=15%]
5 High(Solutiondeliverscompellingvaluetoasubstantialnumberofclearlydefinedcustomers)
4
3 Medium(Teamsuccessfullyarticulatesacrediblevaluepropositiontoanidentifiedcustomerbase)
2
1 Low(Solutionfailstodeliversignificantvalueorcrediblyidentifyacustomerbase)
InitialRating:_________
Ratingafterdiscussion:______________
Comments:5-DEVELOPMENTALSTRATEGY[Overallweight=10%]
5 High(Clearevidenceofintegrationofskillsandeffort)4 3 Medium(Appropriatecollaborativebehaviordemonstrated)2 1 Limited(Evidenceofbadteamworkorcollaborativedysfunction)
InitialRating:_________
Ratingafterdiscussion:______________
Comments:
AudienceViewpointsConsultingArtofScienceLearningResearchReport145
6-FEASIBILITY:Market,technology,capacity,cost,competition,risk/barriersetc.[Overallweight=10%]
5 High4 3 Medium2 1 Low
InitialRating:__________
Ratingafterdiscussion:______________
Comments:7–TEAMWORK/COLLABORATION:Integrationofskills;distributedandcollaborativeeffort[Overallweight=10%]
5 High(Clearevidenceofintegrationofskillsandeffort)4 3 Medium(Appropriatecollaborativebehaviordemonstrated)2 1 Limited(Evidenceofbadteamworkorcollaborativedysfunction)
InitialRating:__________
Ratingafterdiscussion:______________
Comments:
AudienceViewpointsConsultingArtofScienceLearningResearchReport146
AppendixL:Inter-raterReliabilityScoringforCreativitySkillsTest Question ScoreCreativityScore1:1 1CreativityScore1:2 1CreativityScore1:3 1CreativityScore1:4 1CreativityScore1:5 1CreativityScore2:1 0.652CreativityScore2:2 0.652CreativityScore2:3 1CreativityScore2:4 1CreativityScore2:5 1CreativityScore3 CreativityScore5:Impact 0.75CreativityScore5:Feasibility 0.75CreativityScore5:PersonalEngagement 1CreativityScore5:Originality/Uniqueness 1CreativityScore5:ComparativeAnalysis 1CreativityScore5:AncillaryBenefits 1CreativityScore6a:1 1CreativityScore6a:2 1CreativityScore6a:3 1CreativityScore6a:4 1CreativityScore6a:5 1CreativityScore6b:1 1CreativityScore6b:2 1CreativityScore6b:3 1CreativityScore6b:4 1CreativityScore6b:5 1CreativityScore7 1CreativityScore9:Impact 1CreativityScore9:Feasibility 1CreativityScore9:PersonalEngagement 1CreativityScore9:Originality/Uniqueness 1CreativityScore9:ComparativeAnalysis 1CreativityScore9:AncillaryBenefits 1CreativityScore10:What CreativityScore10:How 1CreativityScore12:1 0.652CreativityScore12:2 1CreativityScore12:3 0.652CreativityScore12:4 1CreativityScore12:5 1
AudienceViewpointsConsultingArtofScienceLearningResearchReport147
AppendixM:IdeaClusterCategoriesforCreativitySkillsTest IdeaClustersforPreCreativitySkillsTest,HighSchoolStudents
1.SocialConnectionbetweenneighborhoods2.Access–schedules/distancemostlyrelatedtopublictransportation,accessforthosewithdisabilities3.Safety–ofdriving,publictransportation,walking,biking,cleanlinessoftransportation4.Environment–pollutioncausedbytransportation5.Infrastructure–roads,sidewalks,bikelanes,construction,repair,design6.Alternatives–sharedtransportation,alternativefuelsources,bikes7.Publictransportation–neworrevampedbuses,subways,trollies8.Government–legislationabouttransportation,subsidies,grants,laws,taxes9.Traffic–relatedtodriving10.Economic–costoftransportation11.Misc.–technology,stress,resources,education,advertising
IdeaClustersforPostCreativitySkillsTest,HighSchoolStudents
1.Access–farmersmarkets,grocerystores,location,safety,availability2.Economic–foodcost,costofliving,discounts3.Education&Awareness-classes,campaigns,commercials,information4.School–foodprograms,changelunchplan,classesinschool(homeec,gardening)5.CommunityEffort–locality,charity,communitygardens,buyingorgrowingfoodtogether6.Agricultural–farming,crops,GMOs,chemicals,runoff,pollution,verticalfarming,labeling7.Health–obesity,nutrition,vitamins,etc.8.GovernmentAction9.InfrastructureofFoodSupply–transportandsupplyoffood10.FoodWaste11.Misc.–examples:overpopulation,watersupply,tech,research,jobsEarlySTEMProfessionals
IdeaClustersforPreCreativitySkillsTest,EarlySTEMProfessional
1.Collection(andLoss)ofEarth’swater–salt,rain,storm,desalination,general“runoff”comments2.Recycle/Reusehouseholdwater–greywater,shower,toilet,tap,washingmachine,drinking3.GovernmentAction–taxincentives,regulations,subsidies,laws,mandates,watervalue,emergency4.Conservation(andWaste)–waterfilters,productssuchasshowerheadsor½flushtoilets,communitycontests,householdmetering5.AgriculturalUse6.ResidentialUse–lawns,gardens,nativeplants,“landscape”insomeinstances7.IndustrialUse8.Infrastructure–commentsabout(in)efficiency,transport,piping,concrete,etc.9.EducationandAwareness–includescommentsaboutadvertisingandbranding10.Pollution(andPurification)–e.g.“contamination”and“cleanwatersupply”11.Misc.–energy,composting,oil&gasusage,“foodindustry”,environmentaldetriment(aquaticecosystems),selfishnessofpeople
AudienceViewpointsConsultingArtofScienceLearningResearchReport148
IdeaClustersforPostCreativitySkillsTest,EarlySTEMProfessional
1.Access–farmersmarkets,grocerystores,location,safety,availability2.Economic–foodcost,costofliving,discounts3.Education&Awareness-classes,campaigns,commercials,information4.School–foodprograms,changelunchplan,classesinschool(homeec,gardening)5.CommunityEffort–locality,charity,communitygardens,buyingorgrowingfoodtogether6.Agricultural–farming,crops,GMOs,chemicals,runoff,pollution,verticalfarming,labeling7.Health–obesity,nutrition,vitamins,etc.8.GovernmentAction9.InfrastructureofFoodSupply–transportandsupplyoffood10.FoodWaste11.Misc.–examples:overpopulation,watersupply,tech,research,jobs