a developmental evaluation framework for the city of vancouver solutions lab · 2020-04-07 · a...

A Developmental Evaluation Framework for the City of Vancouver Solutions Lab

Maggie Low

Greenest City Scholar Final Report

September 15th, 2017

ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab

GreenestCityScholarFinalReport,August2017

1

What’stheCityofVancouverSolutionsLab?

TheSolutionsLabisanewapproachtoseekingbreakthrough,transformativesolutionstosomeofthecity’smostcomplex

problems.It’saplacewhereCitystaff,communitymembersandstakeholderscollaboratetodeeplyunderstandcomplexchallenges

fromthepointsofviewofthepeoplemostaffectedbythem,andwherewerapidlyprototypeandtestinnovativeresponsestosee

whatwecanlearnthroughco-creationandsomerisktaking.It’sanexcitingplacewherewedialogueandlistendeeply,trynew

processesandcollaborationtools,andlearnandhavefuntogether.TheCityofVancouverSolutionsLabisusingacombinationof

TheoryUandhuman-centreddesigntogetatbothpersonal/culturalandsystemstransformationsandtoreachbreakthrough

solutions(pleaseseeFigure1.below).

TheCityofVancouverSolutionsLabmodelispartofabroaderpictureof“publicsectorinnovationlabs”(PSIL’s)beingdevelopedin

otherplacesaroundtheworld,especiallyinEurope.PSIL’stakeavarietyofformsandfocusareas;theywillvaryintheirpurpose,

theoryofchange,operatingmodel,methodsandimpact.WhatPSIL’sor“Labs”dohaveincommonistheirstrongaction-reflection

processesandtheyregularlyreviewtheirimpactsandevolvetheirstructure,governanceandfundingmodels,leadership,methods

andsharetheirlearning(Westley,2016).

TheSolutionsLabiscurrentlyinasoftlaunchstage(phase2),withcoremethods,operatingmodel,andappropriatetypesofLab

challengesandteamsallbeingdevelopedbyworkingonfourcomplexchallengesintheLabcontext.FromMaythroughmidAugust

2017,IheldtheGreenestCityScholarpositionandwastaskedwithsupportingtheLabManagerwiththerolloutofLabprocesses

anddevelopinganinitialdevelopmentalevaluationframeworktoevaluatetheimpactoftheLab(moredetailsaboutdevelopmental

evaluationonpg.5).TheSolutionsLabwasinitiatedbyanactiononCollaborativeLeadershipintheHealthyCityStrategy,andhas

onestafflead,aSteeringCommitteeandseniorCitystaffleadership.Todate,communitymembersandstakeholdersonspecific

LabteamshaveconsistedofstaffandvolunteersfromNeighbourhoodHouses,localNGOsandnon-profits,artsorganisations,

TourismVancouverandBusinessImprovementAssociations(BIAs).ThepurposeofthisreportistosharewhattheCityof

VancouverSolutionsLabhasbeendoingandwhy,howwe’vebeenevaluatingouractivitieswithDevelopmentalEvaluationand

basedonourevaluations,whatkindsofquestionscouldwebeaskingforthenextiterationoftheLab.Thehopeisthatthe

SolutionsLabwillofferCItystaffandcommunitypartnersviablewaystoreachdifferentkindsofsolutionstothesustainability

challengesweallface.Morebroadly,solutionsfoundinVancouvercanpotentiallybescaledtocitiesgloballyduetoVancouver’s

leadershippositionandgrowingreachandimpact.

ADevelopmentalEvaluationFrameworkfortheCityofVancouverSolutionsLab GreenestCityScholarFinalReport,August2017

2

SummaryofActivitiesTimelineofSoftLaunch(Phase2)ThesoftlaunchphaseoftheSolutionsLabhasbeenunderwaysinceJanuary2017.ThisreportconsistsofactivitiesandfindingsfromMaytoAugust2017whentheLabManagerandIputtheLabintopracticebyconvening4differentCityproblemsintheSolutionsLabprocessesthatareunderdevelopment.PleasenoteagreatdealofworkwasdonebytheLabManagerandSteeringCommitteebeforemypositionbeganinMay(pleaseseeFigure2)andasummaryofeachLabproblemisinTable1below.

Figure1.(left)AdaptedfromTheoryU(Scharmer,2016)TheoryUisaprocessthatallowsustolearnbynotonlyreflectingontheexperiencesofthepastbutalsolearningfromthefutureasitemerges.Figure1.depictsTheoryUasaprocessthattakesadeepdiveintoaproblemby“lettinggo”ofwhatwethinkweknowandre-emergingwithdifferentsolutionsthanwouldhaveotherwisebeenreached.Human-centreddesignisaprocessusedbydesignersthatinvolvesbuildingempathywiththepeopleyouaredesigningfor.Itstartswiththeusersandendswithnewsolutionsthatarecustommadetosuittheirneeds.


3

Figure2.TimeandKeyActivitiesofSolutionsLabRecentActivities(January2017-August2017)ForeachindividualLabproblem,aLabteamconsistingofCitystaffandcommunitypartners-thoughtfullycuratedbytheLabManagerandLableaders-areledthroughtheSolutionsLabprocessesbasedonTheoryUandhuman-centreddesignprinciples.Theindividualsessionsare1-2daylongworkshopsandhavebeenoccurringover4-6weekperiodsdependingonCitystaffschedules.InthiscurrentiterationoftheLab,eachstageofTheoryU(i.e.Co-initiating,Co-sensing,Presencing,etc.)hashadit’sownsessionwiththesameteammembersinvitedtoparticipateineverystage.Thesessionsarerunbyexperiencedfacilitatorsandheldinlocationsoutside“City”meetingroomssuchasCityStudio,StanleyParkandtheQueenElizabethTheatre.BelowisadescriptionoftheLabsthatarecurrentlyunderwayaswellasotherproblemsthatareintheinitialdevelopmentstages(i.e.-beforeaLabteamiscuratedandbegintheLabprocess).Table1.

Project ConveningQuestion/Purpose Status

EngineeringCustomerServiceLab

HowmightweimprovecustomerexperiencesinEngineeringServices?Wewillusethefrontdeskasafocalpoint,andusewater+sewerpermitsandmajorprojectinterruptionstostreetsasexamples.

Earlyco-creating/prototyping


4

PublicEngagementLab Howmightwebemoreconsistent,collaborative,andvalues-alignedinourpublicengagementworkattheCity?

Movingintopresencing+co-creating

PublicSpace+PublicArtLab

HowmightwebuildacultureofcollaborationbetweentheCityandcommunitypartnerstounlockfurtherpotentialforcreativeandengagedcity-buildingthroughourpublicspaces.

Movingintopresencing+co-creating

CommunityConnections+ResilienceLab

Howmightweincreaseresilience,connectedness,andbelonginginaneighbourhoodwithgrowingdiversityandincreasingdensification?

Co-initiating

WhatisDevelopmentalEvaluation?Developmentalevaluation(DE)isafieldofevaluationmeanttoassistsocialinnovatorscreate-orradicallyadapt-aprogrambasedontrialanderror.DEisdifferentfromothertypesofevaluations;whereassummativeevaluationshelpjudgethevalueofaprogramandformativeevaluationsassistintheimprovementofaprogram,DEevaluateswhatthenextstepsareinthecreationofanemergingprogram.DEasksquestionssuchas:Whatisdeveloping?Whataretheoptionsforthenextiterationoftheprogram?(Cabaj,2014).ThedifferenceisthatinDEtheturnaroundbetweenquestionasking,questionanswering,questioninterpretationanduseforactionoftenhappensinshort,iterativeandongoingcycleswithfocusofdevelopment(Patton,2011,p.232).AnotherwaytothinkaboutDEisas“realtimestrategiclearning”wheretheevaluatorgathersrealtime-datatoinformongoingdecision-makingandadaptationstotheinnovationsbeingimplemented.Patton(2011,p.2)explains:“Theevaluatorisoftenpartofadevelopmentalteamwhosememberscollaboratetoconceptualize,design,andtestnewapproachesinalong-term,ongoingprocessofcontinuousdevelopment,adaptation,andexperimentation,keenlysensitivetounintendedresultsandsideeffects.Theevaluator’sprimaryfunctionintheteamistoinfuseteamdiscussionswithevaluativequestions,thinking,anddataandtofacilitatesystematicdata-basedreflectionanddecisionmakinginthedevelopmentalprocess.”Simplyput,thepersondoingtheevaluatingisoftencloselyinvolvedinthedevelopmentprocessesoftheinnovativeprogram.TokeepdevelopmentalevaluationoftheSolutionsLabsimple,effectiveandtimelythe“What,SoWhat,NowWhat?”inquiryframeworkhasbeenusedbytheSolutionsLabteamtodate(pleaseseeFigure2).Thismodelsimplyappliesbasicevaluative


5

thinkingtoinquireaboutsocialinnovationsanditisanapproachusefultothosenewtoevaluation.Patton(2011,p.232)startswiththefollowingquestionswhenusingDEinanewenvironment:Whatistheinnovationnow?(What?)Whatdotheresultsofinnovationeffortsmeannow?(Sowhat?)Whatdothefindingsrevealaboutnextstepsnow?(Nowhat?)

Figure2.(left)AdaptedfromDevelopmentalEvaluation(Patton,2011)TheSolutionsLabisusingdevelopmentalevaluation(DE)becausetheLabisinadiscoveryphaseanditisconstantlyinmotion.TheSolutionsLabisbeingbuiltaswegoandthereisalotoflearningtodoastheprocessesunfoldinrealtime.Inotherwords,theteamhasbeenintegratingreflectionandadaptationintolabpracticesateverystep.NowweturntothemethodsusedandkeylearningfromPhase2.

Whathavewebeendoingandwhy?AstheSolutionsLabunfolded,DEhelpedtoevaluatethecorepurpose,methods,operatingmodel,andimpactoftheLabfortheCityofVancouveraswellasforitscommunitypartners.OurDEinvolved“trackingemergentandchangingrealities,illuminatingperspectivesandrealities,andfeedbackintomeaningfulfindingsinrealtime”(Patton,2011,p.3).


6

Wehavebeenperformingactionresearchtodothisrealtimelearningandusingthe“What,SoWhat,NowWhat”inquiryframeworktoinformPhase2betweenJuneandAugust2017.Specifically,Iwasintenselyinvolvedinthedeliveryof3outofthe4Labproblems,andbydoingso,wasabletocapturerealtimedatatohelpinformeachphaseoftheLabanddocumentimportantlearningrelevanttootherLabs(pleaseseeFigure3.foranexplanationofhowDEisandwasusedattwodifferentscales).AsperCabaj(2014,p.3),theevaluationdesignandmethodshavebeen“diverse,highlyadaptive,evolvingandlightweightdesignsandemphasisonseekingpatternsacrossexperimentstoinformstrategy.”Specifically,theDEinquiryframeworkof“What?”“SoWhat?”“NowWhat?”hasbeenusedtotestthecoredesignandmethodsusedintheLab(i.e.TheoryU),conveningquestions,Labteamcomposition,facilitatorcombinations,spacerequirements,theoveralloperatingmodelandtheroleoftheLabManager.Wehavebeenpayingcloseattentiontowhatishappeningbefore,duringandaftertheLabsessions(pleaseseeFigure1forarefresherofLabprocesses)tomakesenseofwhatishappening,whyit’simportant(ornot)andifandhowwewanttodoitnexttime.Thistypeofevaluationhasbeenbasedonthefollowingrealities:● Conductedpre-interviewswiththeCommunityConnections+ResilienceLabteamleadsaskingthemtoreflectontheir

expectationsofthelab● Participatedinthedesign,implementation,facilitationandwriteupof3outofthe4labquestions● ParticipatedinthepreparationanddebriefwiththefacilitatorsandLabManagerfor3outofthe4labs● Preparedandcollectedfeedbackfromlabparticipantsattheendofeachsessionandincorporatedthisfeedbackintothe

planningofnextlabs● Assistedwiththewriteupof3labreports● Tookdetailednotesandpicturesduringthelabsessions● ConsultedwithDEexpert,MarkCabaj,andreadrelevantDEliterature

Muchhasbeenlearnedontheoperationalscale(bothwithineachLabandacrossLabs)andwearestartingtoprobeandgleaninsightsintowhatthelargerimpacttheLabcouldhavefortheCityasawhole.BelowisasummaryofkeylearningfromtheLabactivitiessofar.PleaseseeAppendixAforasamplefeedbackformusedattheenddayforeachLabsession.


7

Scale1:WithinandBetweeneachofthe4LabsDEwasusedtoevaluateeachphaseofindividualLabs(i.e.EngineeringCustomerServiceLabORPublicEngagementLab)bycollectingrealtimedataandeitheradaptingtochallengesandquestionsastheyaroseorbytakingnoteofoutcomesofourdecisionsinoneLabandusingwhatwelearnedinthedesignofotherLabs.ThegraphictotheleftdepictsanexampleofonephaseofaLab(Co-sensing)thatcanbeevaluated.AllotherphasesoftheLabwerealsoevaluatedusingDE.Example#1:TrackingwhoandwhenLabteammembersattendtheLabsessionhasbeenaninterestingchallenge,forexamplewhennewpeoplewereinvitedtoparticipateintheco-sensingpartoftheLabwithoutbeingpartoftheco-initiatingprocess.Initiallyitwasthoughtthenewparticipantswouldbeanunhelpfuldistraction.However,withminortweakstothe“re-cap”sessiononthefirstdayoftheco-sensingsession,newlabparticipantswerebroughtuptospeedandaddednewperspectivesandenergytotheroom.

Scale2:TheSolutionsLabprocessesasawholePreliminaryideasabouthowtomeasuretheimpactoftheLabhavebeguntosurfacefromseveralplacesincludinglessonslearnedfromDEineachLab,debriefmeetingswithfacilitators,reflectionsfromSteeringCommitteemembersandmeetingswithseniorCitystaff.AtthisphaseitwouldbeusefultotakestockofwhatwehavelearnedanddevelopaconcreteevaluativeframeworktomeasuretheimpactoftheLabintermsofbreakthroughs,transformationsandrelationships.Pleaseseebelowfordetaileddefinitions.Example#2:ThroughouttheLabprocess,thetimecommitmentrequiredbyCitystaffandcommunitypartnershasbeensubstantial.WeshouldaskandtestwhetherpartsoftheLabcanbeusedfordifferentkindsofproblems.DoeseachproblemhavetogothroughtheentireUtoreachdifferentsolutions?

Figure3.DevelopmentalEvaluationusedatdifferentscales


8

KeyLearningfromPhase2oftheSolutionsLab:

● Whoisontheteamreallymatters.○ Theteamleadsneedtobe“pioneers”.Thesearepeoplewhoonlyneedtoknowthewhy,andnotnecessarilythe

“what”orthe“how”.Theteamleadswillparticipateintheprocessinvariousways,frombeingastrongervoiceinthe

roomtomoreofaquietyetencouragingvoice.Differentvariationsofthisspectrumwillwork,whatisimportantis

thattheteamleadsareadvocatesoftheprocessandarewillingtoparticipatewithopenmindsandhearts.○ Diversityreallymatters!Participantsmadeveryclearthatdiverseperspectives,educationalbackgrounds,worldviews,

andethnicitiesareextremelyimportanttoanauthenticLabexperience.TheLabManager,supportandteamsleads

needtotakeaccountforthistothebestoftheirabilityfromtheverybeginningoftheprocess.○ Findingthe“pioneer”staffleads,andtheproblemswiththerightamountofurgencyandcomplexitytothem,takes

timeandintentiontogetitright.

○ Teammembersdon’tnecessarilyneedtocommittothewholeprocessrightaway;theycansayyesatthebeginning

butthingschange.Itwasusefultoaskforeachperson’scommitmentattheendofthe“Co-initiating”workshopasall

participantsexpressedinterestandcommitmentingoingfurtherintheprocesstogether.● Timeisprecious.

○ WearelearningtodesigntheLabprocesswithpeople’stimeandschedule’sinmind,whilestillfacilitatinga“deep

dive”intotheproblem(i.e.designsprintand/orshortersessionoverafewweeks,allowingforacertainamountof

coming/goingoflabteammembersduringsessions).ThemoreintensedesignsprintformatusedinoneoftheLab’s

allowedustogothrough(almost)thewholeprocessandcometoconceptsforprototypes.Whilestressful,itwas

importantforustodoitthiswaybecausewelearnedusefulinformationaboutwhenandhowtonarrowthescopeof

theproblem,andhowchallengingitcan(andwillbe)togettothe“Co-embodying”phase.○ Planningaroundpeople’stimeandschedulesisalwayschallenging;duringthesummermonthswe’vebeentryingto

workaroundvacationtimeandstilltryingtohitthesweetspotoftiming(i.e.-1,2or3weeks)betweeneachphase

intheLabprocess.Iftoomuchtimelapsesbetweeneachsession,doestheteamlosesteamandinterest?Sofarthe

answerisno,theLabteamsseemgenuinelycommittedtotheentireLabprocess.

● Productivetension?(PressurecookermetaphorfromMarkCabaj)○ Thecook(beingthefacilitationteam)hascontroloftheturningtheheatupordown.Ifthepressuregoesbeyondthe

carryingcapacityofthevessel,thepressurecookercanblowup.Ontheotherhand,withnoheatnothingcooks.We

don’twanttoeliminatestress/tensioncompletely.Wehavebeenlearningtomanagethesetensionsastheycomeup

intheLab:


9

■ Outcomesversusprocess:Thereisastrongtensionbetweenprocessandoutcomes;weallknowachieving

tangibleoutcomesisimportantforamunicipalgovernment,sohowmuchtimecantheLabfocusonthe

“deepdive”processwithoutlosingthegoalofcomingtotestableprototypes?■ Consensusversusdebate:howtobalancecollegialandopenspacewithdiscomfortandconflictingviews■ Supplyversusdemandoftime:howmuchtimeistoomuch,ortoolittle?Shouldwebespendingmoretime

unpackingthelearningjourneys?● Everyonelovesalearningjourney.

○ Maybenoteveryonelovesthispartofco-sensing;however,it’sbeenaninvaluablepartoftheLabprocessintermsof

pushingCityStafftoseetheirworkthroughthelensofthosemostimpactedbyit.It’smorethan“consultationwith

stakeholders”;it’saboutlistening,observingandaskingquestionstogainanewperspectiveontheproblem,and

thentakingthoseinsightsbacktotheideationstage.● Facilitatorsarekey,soweneedmore.

○ DifferentconsultantshavedifferentabilitiestofacilitatethecurrentSolutionsLabframeworkandbringtheirown

specifictool-kitstoit.Therecurrentlyseemstobeenoughcapacityintheconsultantecosystemtocreativelyfacilitate

theUprocess;howevertheremaybefewerconsultantsinVancouverthatcanactually“build”avarietyofdifferent

typesofprototypes.● LabManagerasaconvener,withalittlebitofhelp.

○ TheroleofLabManagerhasbeentoconvene,organize,coordinate,andhelpfacilitatetheLabinitsentirety.Thisrole

hasbeenaboutmaintainingrelationships,bridgingtoimplementation,andensuringasafespaceinwhichLabprocess

canoccur.Facilitationconsultantscanthenprovidemoreuncomfortablepushesifneeded,withoutriskingthese

relationships.TheLabManagerholdsacertainsenseofaccountabilityto/fromteamparticipants.Thismodelhas

workedwellsofar.

○ Having4problemsintheLabatonetimehasbeenmanageableandcancertainlycontinueaslongasthereis

supportstaff(i.e.-eachlabtakesquiteabitoftimetoplan,unfoldandwrapup).

● It’sallaboutthefood...andspace.○ It’sbeencrucialforLabstobeoutofregularCoVmeetingspacesandcontextsandthereforewe’vebeenmoving

aroundcityintodifferentneighbourhoods,accessingoutsidespace(i.e.THNKHome,ParkBoard/StanleyPark,Art

Starts,CityStudio,SlocanParkFieldHouse).Weprovidenicefood,warminvitations,andpersonaltouchesasmuchas

possible.Peopleappreciatethisasthisdemonstrationofcare,andhonouringpeople’sparticipationhasbeen

importanttocreatingagraciousspace.○ Thislevelofdetailhasalsobeenlogisticallychallenging(i.e.carryingboxesofsuppliesonourbikes).Itisworthwhile


10

havingapermanentLabspace(thatisbrightandwithakitchen)ifitcanprovidean“outofCoV”experience.● WhatmakesaLabaLab?

○ WeneedmoretimeintheLabtofigureout“whatisaLabandwhatisnotaLab”?It’sanimportantquestionthat

keepscomingforwardfromtheLabSteeringCommitteeandseniorCitystaff.Wemayneedtothinkabouthowwe

canapplywhatwehavelearnedsofarelsewhereineasy/simpleways.Canalabbeone½daysessionthatoffersa

creativepush?Doesitneedtobeaseriesofin-depthworkshopstogetatbreakthroughsolutions?

○ DevelopmentalEvaluationhasbeenhelpfultogetustoshapeandrunwhatisseeminglikeawellroundedLab

process,thoughastheLabmovesintoitsnextiterationotherDEinquiryframeworksmaybenecessarytotrackthe

fullimpactoftheLabintermsofbreakthroughs,transformationandrelationships.Howdowemeasuretheoverall

impactoftheSolutionsLabfortheCityofVancouver?Thiscouldincludefollowing“keyinformants”throughtheLab

processfrombeginningtoend.PleaseseeTable2.below.

What’snextfortheSolutionsLab?

AstheLabcontinuesintothefall2017,3outofthe4LabswillmoveintotheCo-creationandCo-embodyingsessionsandpotentially

2newproblemswillbetakenonbytheLab.Astheproblemsbecomemorecomplex,itwillbeimportanttocontinuetoadaptthe

overallpurposeoftheLabwiththeLabManager,SteeringCommittee,andDeputyCityManager.AstheLabunfoldedinpractice,

DEhashelpedmakethecasethatmeasuringtheLabintermsofbreakthroughs,transformationsandrelationshipsisclearand

simple,andalsoconveysambitionandastrongcalltoaction.Howmightweunderstandandmeasuretheseimpacts?

MuchoftheinitialthinkingabouthowtomeasuretheoverallimpactoftheLab,oralsoknownastheLab’s“theoryofchange”took

placebeforemyGreenestCityScholarpositionbeganinMay2017.PleaseseeAppendixAforaninitiallistof“UnitsofAnalysis”to

useaspotentialindicators.Todate,thefollowingdescribespossiblewaysofmeasuringtheoverallimpactoftheLabintermsof

breakthroughsolutions,transformation,andrelationships:

Breakthroughs(akagettingtodifferentkindsofsolutions):Innovation,risk-taking,experimentation,action,complexproblems,prototyping,failure-as-learning,creativedisruptionTransformations(akapeopleshowingupdifferentlyandseeingthemselvesasagentsofchange;transformingthecultureandsystemsthatwe’reworkingwithin):Systematic,personal,leadership,action/reflection,developmental,intrapreneurship,newmetaphors+stories


11

Relationships(akabreakingdownsilos):Diverseperspectives,collaborative,empathy,newinsights,wholesystemintheroom,deeplisteningPotentialmethodstomeasurethethreeareasofimpactinclude:

Table2.

Method PotentialQuestions

UserProfileWhatwillyouwanttoknowabouttheoverallimpactoftheLab?

Identifyprimaryusersoftheevaluation

andcompleteuserprofileasperCabaj

(2014).

Organizeandprioritiseusercasesto

ensuretheevaluationispurposefuland

relevant.Usetheuserprofiletoorient

DEtotracklargerimpactof

breakthroughs,transformationsand

relationships.

1)Howwouldlifebebetterifthisprojectworkedwell?

2)Whatisitaboutthelabthatyouthinkwillmakeadifference?

3)Howwillthelabovercomelikelychallengesandbarriers?

PleaseseeAppendixCforotherpotentialinterviewquestionsto

completetheuserprofile.

Pre+postinpersoninterviewsWhatarecoreteammemberslearningbygoingthroughtheLab?Arestaffseeingthemselvesin

Conductpreandpostinterviewswith

coreteammembersforSolutionsLab

problems.Preinterviewswillaskeach

intervieweetoreflectontheirmindset

goingintothelab.Postinterviewswill

askeachintervieweetoreflectonwhat

theyfeelwouldnothavehappened

withoutthelab.Thepreandpost

interviewsareintendedtogleana

deeperunderstandingofhowthecore

methodsusedwithinineachlabare

PotentialPreInterviewQuestions

Howdoyoucurrentlyperceive/understandtheproblem?

WhatareyourhopesandfearsfortheSolutionsLab?

WhataresomeofthebarrierstoinnovationfortheCity?(Tryto

getpeopletoseetheyhavemoreagencyinthesystemor

somethingthatgetsthemtothinkaboutroleinbiggersystem.

Provideanexample…)

HowdoyouthinktheSLabwillhelpwiththisproblem?

PotentialPostInterviewQuestions


12

moreexpansiveways?

working,whyorwhynot.WriteupallreflectionsandpreandpostinterviewdatainexcelsheettotrackandcodethepotentialchangestakingplacethroughouttheSLabprocesses.

Howdidthelabhelpchangethisperception?Inyouropinion,whatwasthebiggestchallengeyouovercameinthisprocess?Whatwouldyouhavedonedifferently?Whatwouldhavenothappenedwithoutthelab?

PreandpostLabonlinesurveysPromptstogetatmindsetsbeforecomingintothelabaswellasmindsetsaftertheLabexperience.

PrepareashortonlinesurveytosendtoallLabteammembers1-2daysbeforetheLabbegins.ThiswillgiveyouasenseofwhattheteammembersknowandfeelabouttheLab,andhowtheyperceivetheconveningquestion.Sendthesame,orsimilaronlinesurveytoeachLabteammemberafterthefullLabhasbeencompletedtogleanifhowtheirunderstandingoftheconveningproblemchangedthroughouttheLabprocessandwhy.

ThefollowingpreLabonlinesurveyquestionswereusedfortheCustomerServiceinEngineering:Inyouropinion,howimportantisittoimprovecustomerserviceinEngineering?Inyourexperience,whatarethetopthreeissuesor‘areasoffocus’forimprovingcustomerserviceinEngineering?Whatarethemostusefulandeffectivetools,equipment,processes,etc.thatareavailabletoyou,andenableyoutoprovidegreatcustomerservice?Whatarethemostfrustratingtools,equipment,processes,etc.thatgetinthewayofprovidinggreatcustomerservice?WhenEngineeringhasimprovedourcustomerservice,whatdoesthatLOOKandFEELliketoyou?WhataresomeofthebarrierstoinnovationinEngineering?WhataresomeofthebarrierstoinnovationattheCityingeneral?


13

BiggerQuestionsfortheSolutionsLab

BelowarethemainquestionsIwouldrecommendtheLabfocusoninitsDEworkmovingforwardintothePhase3andbeyond:

Labpurpose● DoestheLabneedanidentity?● Whatarethebasicrequirementsforworktobedonein“theLab”?(i.e.itmustbeacomplexproblem,itmusthavea

committedstaffleadwhohastheproblemontheirworkplan.)● HowcantheSolutionsLabbesthelpachievecitypriorities?● What,ifany,partsofspecificLabprocessescanbeleveragedindividuallytobenefitotherprojectsandprograms?● Howcanweiterateourtheoryofchange?

Impact● Tomeasureimpactshouldweincreasethesamplesize?(i.e.-usedatafromthenextiterationofLabs)● Howcanwebestmeasurebreakthroughs,transformations,andrelationships?

Methodsandprocess● WhataretheconditionsthatmakeTheoryUappropriateornottotakeadeepdiveintotheproblem?● WhatpartsofTheoryU,human-centereddesign,andotherpotentiallabmethodscanbeusedindependentlybydifferent

departmentsandforwhichproblems?

● WhatLabmethodsaremostappropriate,highimpactandengagingindifferentsituationsandfordifferentproblems?● Re-evaluatetheintentionaldifferencesbetweenLabmodels-designsprintversusoveralongerperiodoftime-what

workedwellandwhatdidn’t?● Howdoweensureprototypesdon’tdieoff?● Howcanwesupportrisk-takinginuncertainproblemspaces?

● HowdoweensurethatsuccessfulprototypesareintegratedintoCityactivitiesandoperationsineffectiveways?

Typesofconveningquestionsandteams● Whattypesofproblemsarecomplexenough?● WhatarethequalitiesofaLabteamthatneedtobeinplace?

Space+aestheticexperience● WouldapermanentLabspacecreatethesameatmosphere(i.e.-helpLabteam“showupdifferently”)

OperatingModel● WhatarethebenefitsandlimitstothemodelofaLabmanagerandanecosystemofconsultants?


14

● Whatdoessuccessionplanningneedtolooklike?● Canwecommunicate“successstories”oftheLabinternallyandexternallytogeneratebuy-inandunderstandingaboutthe

approach?


15

References

Cabaj,M.(2014).WhatWeKnowSofarAbouttheStrategyContinuum.HeretoThereConsulting.

Hwalek,M.,&Williams,M.G.(2011).Thereal-timeevaluationmemo:Atoolforenablingevaluativethinkingandlearningin

foundationsandnonprofits.TheFoundationReview,2(3),25-35.

Patton,M.(2011).DevelopmentalEvaluation:ApplyingComplexityConceptstoEnhanceInnovationandUse.London:TheGuilfordPress

Scharmer,O.(2009).LeadingfromtheFutureAsItEmerges:TheSocialTechnologyofPresencing.Berrett-KoehlerPublishers

Westley,F.,Laban,S.,Rose,C.,McGowan,K.,Robinson,K.,Tjornbo,O.,Tovey,M.(2016).SocialInnovationLabGuide.Waterloo

InstituteforSocialInnovationandResilience.


16

APPENDIXA-ExampleFeedbackFormfromDay5ofEngineeringLab

EngineeringCustomerServiceLabDateandLocation

Thankyouforyourblood,sweatandtearsoverthelasttwoweeks.Beforeyougo,weneedyourfeedbacktohelpusdeterminetheimpactoftheSolutionsLabs,andplansuccessfulLabsinthefuture.1.I’velearnednewmethodsand/orIhavenewtoolstobringbacktomywork/team?

StronglyDisagree

Disagree SomewhatDisagree

Neutral SomewhatAgree

Agree StronglyAgree

2.Ihavedevelopednewrelationshipsand/ordeepenedexistingrelationshipsbecauseoftheSolutionsLab.

StronglyDisagree



Agree StronglyAgree

3.Asalab,wecametosolutions/prototypesthatwouldnothaveotherwisebeencreated?

StronglyDisagree



Agree StronglyAgree

4.WhatwasthebiggestpersonalchallengeforyoutoovercomethroughouttheSolutionsLabprocess?5.DoyouunderstandtheproblemofcustomerserviceinEngineeringdifferentlythanwhenyoustarted?Ifso,how?6.WhatadvicedoyouhaveforusforfutureLabs?


17

APPENDIXB-Evaluation“UnitsofAnalysis”● Fundingleveraged● Prototypestested● Professionalandpersonaldevelopmentopportunities–teammemberslearnednewskill,teamworkimproved,verticaland

horizontaldevelopment,moreengagedincity,broughtwholeself● Clarified/honedroleforlabwithinbroadercitycontext● Successfullyfailed● Generatedexcitementinternally● Didn’tgetboggeddowninhierarchy● Newmethodstried● Communityrelationshipsstronger.Builttrust.Sharedpower.● Collaboration● Breakthroughs● Internalcoverage/storytelling● Personaltransformation;signsofsystemstransformation● Builtandsupportednetworkofprocessexperts● Eachprojectteamcansay:wegotabituncomfortableandtriedsomenewthings.Welearnedsomenewmethodsthatwecan

bringbacktofutureprojects.We’reastrongerteam.We’vebuild/strengthenedcommunityrelationships.● Whatisonethingtheyareworkingonthatwillrequiretheygrowtoaccomplishit● Howaretheyworkingonit● Whoelseknowsandcaresaboutit● Whydoesthismattertothem


18

APPENDIXC-InterviewQuestionstoCompletetheUserProfile

1. Whatbenefitsmightyouexpecttosee?2. Whatarethecurrentbarrierstoachievingthis?Howwillthelabovercomethese?3. Howwouldlifebebetterforstaff,stakeholders,thecityifthisprojectworkedwell?4. Whatisitaboutthelabthatyouthinkwillmakethedifferenceforthem?5. Whatelsewillbeneededtoachievetheseresults?6. Whatmightbesomenegativeimpacts?Howmighttheycomeabout?Howmightwereducetherisksofthoseimpacts?

a developmental evaluation framework for the city of vancouver solutions lab · 2020-04-07 · a...

Documents