the five dysfunctions of a data engineering team
Post on 22-Jan-2018
944 Views
Preview:
TRANSCRIPT
TheFiveDysfunctionsofaDataEngineeringTeam
1/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Chapter1
TheFiveDysfunctionsofaDataEngineeringTeam
2/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
3/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
85%ofBigDataprojectsfailtogetintoproduction
Source:http://tiny.bdi.io/gartnerfail
Failure
4/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
I'dtrainatcompaniesandseefailuresattheirmid-point
Ittookawhiletoseethepatterns
Ittookmoretimetofigureoutthemostcommonpatterns
BigdataonlyamplifiesexistingproblemsIfyoubarelygetbywithsmalldata,you'llhavebigproblemswithBigData
Youcanbesuccessfulbyavoidingtheseproblems
Why?
5/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
6/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
DBADefinition-SomeonewhoseonlyprogramminglanguageisSQL
Thisincludesdatawarehouse,SQLDevelopers,etc
BigDataisnotanextensionorthelogicalextensionofdatawarehousing
It'smuchmuchmorecomplexhttp://tiny.bdi.io/complex
It'snotjustaskillsgap;it'sanabilitygaphttp://tiny.bdi.io/abilitygap
AllDBAs
7/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
BigDataiscomplexhttp://tiny.bdi.io/complex
Beginnersneedtobegivethetimeandresourcestolearn
Ittakesatleast6monthsforabeginnertobecomeproficient
Asyoulookatsuccessfulcasestudytalks,theyleaveout
ExpertresourcesprovidedStartingproficiencyoftheteamTotaltimeused
SetUpForfailure
8/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Schemaproblemsdon'tmanifestimmediately
Takes6-12monthstosee
Youcan'tlaydownPBsofdataandchangeit
Datapipelinesneedtochangedataformats
Whichroletypicallyhasthisskill?DBAs(Ididn'tsaynoDBAs-IsaidnotjustDBAs)
NoOneUnderstandsSchema
9/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
AprojectveteranissomeonewhohasputaBigDataordistributedsysteminproduction
BeginnerstodistributedsystemsandBigDataarethesourcesoftheworstabominations
Averagetimelostis1-2manmonths
Verydifferenttowhiteboardanderasethancodeandrewrite
OnlyaprojectveterancancritiqueawhiteboardedarchitectureRememberit'sprogrammingandoperations
NoVeterans
10/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Youcan'tgofrom0toBigDataallatonce
Youreallycan'tgofrom0totheholygrail
Yourteamneedsthetimetogofrombeginnerstointermediatetoadvanced
YouneedtobuildmomentumfirstProjectswithoutmomentumgetcanceled
TooAmbitious
11/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
WhyWorryTheDysfunctionsWhattoDo?
TheFiveDysfunctionsofaDataEngineeringTeam
12/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
TakeanhonestevaluationoftheteamSkillsAbilitiesUsecaseResources
Doestheteamhaveaskillsgap?
Doestheteamhaveanabilitygap?http://tiny.bdi.io/abilitygap
DoesthisSoundLikeYourTeam?
13/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Dataengineeringteamsneedtobemultidisciplinary
http://tiny.bdi.io/detbook
DataEngineeringTeams
14/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Someteamssaytheydon'tneedhelpTechnicalpeoplethinkit'snotneeded(smalldatamentality)Admissionoffailure
Veryimportanttotakeanhonestlookattheteam
Training
ConsultingVeryimportanttogetacompanywithagoodtrackrecord
MentoringOngoinghelpforthetechnicalandbusinessteams
GettingHelp
15/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
100
80
60
40
20
0First ReleaseTeam Creation Project Start Second Release Nth Release
Phase In Project
Percent of Blame
Management Development Operations
WhenThingsFail
16/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Early
Nevertoolatetofix,butfixingwillbemuchmore
costly
WhenShouldYouFix?
17/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
Current:Instructor,ThoughtLeader,MonkeyTamer
Previously:CurriculumDeveloperandInstructor@ClouderaSeniorSoftwareEngineer@Intuit
Covered,ConferencesandPublishedIn:GigaOM,ArsTecnica,PragmaticProgrammers,Strata,OSCON,WallStreetJournal,CNN,BBC,NPR
SeeMeOn:http://www.jesse-anderson.com@jessetandersonhttp://tiny.bdi.io/linkedinhttp://tiny.bdi.io/youtube
AboutMe
18/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f
top related