TheFiveDysfunctionsofaDataEngineeringTeam

1/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Chapter1

TheFiveDysfunctionsofaDataEngineeringTeam

2/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

WhyWorryTheDysfunctionsWhattoDo?

TheFiveDysfunctionsofaDataEngineeringTeam

3/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

85%ofBigDataprojectsfailtogetintoproduction

Source:http://tiny.bdi.io/gartnerfail

Failure

4/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

I'dtrainatcompaniesandseefailuresattheirmid-point

Ittookawhiletoseethepatterns

Ittookmoretimetofigureoutthemostcommonpatterns

BigdataonlyamplifiesexistingproblemsIfyoubarelygetbywithsmalldata,you'llhavebigproblemswithBigData

Youcanbesuccessfulbyavoidingtheseproblems

Why?

5/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

WhyWorryTheDysfunctionsWhattoDo?

TheFiveDysfunctionsofaDataEngineeringTeam

6/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

DBADefinition-SomeonewhoseonlyprogramminglanguageisSQL

Thisincludesdatawarehouse,SQLDevelopers,etc

BigDataisnotanextensionorthelogicalextensionofdatawarehousing

It'smuchmuchmorecomplexhttp://tiny.bdi.io/complex

It'snotjustaskillsgap;it'sanabilitygaphttp://tiny.bdi.io/abilitygap

AllDBAs

7/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

BigDataiscomplexhttp://tiny.bdi.io/complex

Beginnersneedtobegivethetimeandresourcestolearn

Ittakesatleast6monthsforabeginnertobecomeproficient

Asyoulookatsuccessfulcasestudytalks,theyleaveout

ExpertresourcesprovidedStartingproficiencyoftheteamTotaltimeused

SetUpForfailure

8/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Schemaproblemsdon'tmanifestimmediately

Takes6-12monthstosee

Youcan'tlaydownPBsofdataandchangeit

Datapipelinesneedtochangedataformats

Whichroletypicallyhasthisskill?DBAs(Ididn'tsaynoDBAs-IsaidnotjustDBAs)

NoOneUnderstandsSchema

9/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

AprojectveteranissomeonewhohasputaBigDataordistributedsysteminproduction

BeginnerstodistributedsystemsandBigDataarethesourcesoftheworstabominations

Averagetimelostis1-2manmonths

Verydifferenttowhiteboardanderasethancodeandrewrite

OnlyaprojectveterancancritiqueawhiteboardedarchitectureRememberit'sprogrammingandoperations

NoVeterans

10/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Youcan'tgofrom0toBigDataallatonce

Youreallycan'tgofrom0totheholygrail

Yourteamneedsthetimetogofrombeginnerstointermediatetoadvanced

YouneedtobuildmomentumfirstProjectswithoutmomentumgetcanceled

TooAmbitious

11/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

WhyWorryTheDysfunctionsWhattoDo?

TheFiveDysfunctionsofaDataEngineeringTeam

12/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

TakeanhonestevaluationoftheteamSkillsAbilitiesUsecaseResources

Doestheteamhaveaskillsgap?

Doestheteamhaveanabilitygap?http://tiny.bdi.io/abilitygap

DoesthisSoundLikeYourTeam?

13/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Dataengineeringteamsneedtobemultidisciplinary

http://tiny.bdi.io/detbook

DataEngineeringTeams

14/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Someteamssaytheydon'tneedhelpTechnicalpeoplethinkit'snotneeded(smalldatamentality)Admissionoffailure

Veryimportanttotakeanhonestlookattheteam

Training

ConsultingVeryimportanttogetacompanywithagoodtrackrecord

MentoringOngoinghelpforthetechnicalandbusinessteams

GettingHelp

15/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

100

80

60

40

20

0First ReleaseTeam Creation Project Start Second Release Nth Release

Phase In Project

Percent of Blame

Management Development Operations

WhenThingsFail

16/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Early

Nevertoolatetofix,butfixingwillbemuchmore

costly

WhenShouldYouFix?

17/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f

Current:Instructor,ThoughtLeader,MonkeyTamer

Previously:CurriculumDeveloperandInstructor@ClouderaSeniorSoftwareEngineer@Intuit

Covered,ConferencesandPublishedIn:GigaOM,ArsTecnica,PragmaticProgrammers,Strata,OSCON,WallStreetJournal,CNN,BBC,NPR

SeeMeOn:http://www.jesse-anderson.com@jessetandersonhttp://tiny.bdi.io/linkedinhttp://tiny.bdi.io/youtube

AboutMe

18/18Copyright©2016SmokingHandLLC.AllrightsReserved.Version:ef81f3f