inforum 2016 keynote: data and information quality

Change your Organization's Culture to Make Data and Information Quality a Part

of it’s DNA

inForum 2016Perth, Australia

September 13, 2016

Jay ZaidiManaging Partner

My Books, About Me, and Contact Details

Contact DetailsEmail – jayzaidi@alydata.com

LinkedIn - http://www.linkedin.com/in/javedzaidiWeb – http://www.alydata.com/

2

My books on “Data-Driven Leadership” launched worldwide on Amazon and Kindle in July 2016.

About Me

Founded AlyData after two decades in the industry. In my last corporate role I reported directly for five years to the Chief Data Officer of the largest financial services company in the world. Worked for PriceWaterCoopers LLC, Commerce One, and DOW Chemical Company prior to that.

AGENDA

Part 1 – Age of DataPart 2 – Quality is Job 1Part 3 – There is a Trust Deficit that must be OvercomePart 4 – Change Culture or Get DisruptedPart 5 – Call to Action

3

Our World’s Being Turned Upside Down

4And You Should Think About What This Means to You!

The Fourth Industrial Revolution

“We stand on the brink of a technological revolution that willfundamentally alter the way we live, work, and relate to one another. Inits scale, scope, and complexity, the transformation will be unlikeanything humankind has experienced before. This is the FourthIndustrial Revolution or the digital revolution that has been occurringsince the middle of the last century. It is characterized by a fusion oftechnologies that is blurring the lines between the physical, digital, andbiological spheres.” – Klaus Schwab, Executive Chairman of The WorldEconomic Forum

5

I’ve labeled the Fourth Industrial Revolution the “Age of Data.”

Massive Disruption Is Happening In Every Sector - Your Company May Be Next

6

The Common Thread Across Disruptors are Data and Insights! High Quality Data is required for best insights.

5 Pillars of the New Business Model1. VarietyandDecentralization:Social,Mobile,Analytics,andCloud

(SMAC)driveoperations2. BetterInsights:Nearrealtimeinsightsfordecisionmaking,risk

management,andtogaincompetitiveadvantage3. Agility:TransformationoftheoperatingmodelfromSDLCtoAgileand

introductionofautomatedprocesses4. Transparency:Sharingeconomyrequiresasharingculture.Changein

teamdynamicstobecomemoretransparentandsharedataandalgorithms.

5. Innovation:Innovateusingdata,people,algorithms,andprocess.Newareassuchasartificialintelligence(AI),deeplearning,intelligentconversationengines,speechrecognition,and imageandpatternrecognition.

7

The Intellectual Capital of this new world is Algorithms, Data, and People.

A New Leadership Paradigm -Leadership 2.0

8

Leaders and aspiring leaders must become “data savvy” and pivot on “data” not IT.

Leadership1.0 Leadership2.0

Quality Is Job 1

9

10

Quality is a State of Mind and Has To Be Incorporated Into All Data Processing Steps.

• Inthe1980’sJacquesNasserwasCEOofFordandintroducedthisslogan

• WantedtotransformFordintotheleadingconsumerproductscompany

• Thisinitiativechangednotjusttheculturebutthequalityoftheendproducts

• ResultedinaUS$300Millioninreducedscrap,rework,andnonvalueaddedactivities.Thisisequivalentto$754Millionintoday’sdollars–asignificantsavings.

11

https://www.youtube.com/watch?v=X3ecGr_-2vQ

https://www.youtube.com/watch?v=X3ecGr_-2vQ

12

https://www.youtube.com/watch?v=Sx_6OoFmycohttps://www.youtube.com/watch?v=Sx_6OoFmyco

https://www.youtube.com/watch?v=Sx_6OoFmyco

4 Eye Opening Facts

• DataQuality:Atleast6%to10%ofIToperatingbudgetwastedduetore-workandinefficientprocessing• DataWrangling:70%to80%ofdataprocessingtimeandcostisassociatedwithdatawrangling• DarkData(acquiredbutneverused):85%ofdataacquiredisn’tusedforanythingofvalue• Metadata(context):Inabilitytofinddata,understanddatasemantics,anddatarelatedrulesresultsinmassiveinefficiency

13

Companies need data quality founders, evangelists, and data quality owners across all departments.

Companies Require Small and Big Data To Succeed

Traditional(SmallData) Data-driven(BigData)Highlystructureddata Structured, Unstructured,Semi-structured

Pre-defineddataschemas FlexibledataschemasPre-defined datamodels(schemaonwrite) Undefineddatamodels(schemaonread)Relationaldatabase managementsystems HadoopandNoSQL datastores

Silosofdata BigDataLakes(consolidateddatasets)Performance andscalabilitylimitations Infinitescaling

Mostlyonpremise data Highlydecentralizeddata(Cloud)DataMiningandBusinessIntelligence Predictive,Prescriptive Analytics,DeepLearning

andArtificialIntelligence

14

The introduction of Big Data into the “Data driven” business model requires a culture change - new data quality and project management

skills, new execution capabilities, and agility at its core.

There Is A Trust Deficit That Must Be Overcome

15

Trust Deficit

16

Business transactions are built on trust. Unfortunately, there is a trust deficit today and it must be addressed.

Producers and Consumers Operate On Trust

17

Government agencies define standards and policies that producers must follow to ensure quality and transparency.

Labels

Food labels present nutritional and other information to help consumers make safe and healthy food choices. Some labelling information is mandatory, while others are voluntarily added by manufacturers. Labelling must include a list of ingredients and food additives, as well as any potential allergens. A nutrition panel outlining levels of key nutrients is also required.

We use food labels:

• For health reasons• To avoid particular ingredients or food additives• For personal beliefs, such as avoiding genetically modified foods or

foods containing animal products or to buy items grown locally.• Food labels must tell the truth and include:• Name or description of the food• Nutrition information panel• Ingredient list• Percentage labelling• Food additives• Country of origin• Food recall information• Directions for use and storage• Information for allergy sufferers• Legibility requirements• Date marking.

When Was the Last Time You Were Provided a Label with your Data Sets?

18

Seems logical. But it’s never done in the industry. Shouldn’t it?

Data Set Labels

Data Set labels present information that describes the content of the data set and other information to help consumers understand what’s in it and to make choices on its usage. Some labelling information should be mandatory, while others are voluntarily added by producers. Labelling must include a list of key data ingredients and any data enrichment performed on it (additives), as well as any potential transformations (allergens). A data panel outlining levels of sensitive and personally identifiable data should also be required.

We use data set labels:

• for business transaction reasons• to be aware of particular sensitive or personally identifiable data so that we can handle them with care

Data labels must tell the truth and include:

• Name or description of the data elements in the data set• Data consumption information panel• Data element list• Data quality labelling• Any enrichments• System(s) of origin• Directions for use and storage• Information for special handling• Date marking.

19

An Inspection Regime Is Needed

One must trust but verify quality. Companies must implement an inspection regime to audit data quality at every step.

Data Quality Process Flow

20Repeatable Data Quality Processes must be Implemented.

Australian Bureau of Statistics Quality Framework

21

Suggestedquestionstoassessinstitutionalenvironment:• Whichorganization(s)hassuppliedthedata?• Whatsortoforganizationisthis(e.g.,public,commercial,non-governmentorganization)?• Underwhatauthorityorlegislationwerethedatacollected?• Whatproceduresareinplacetoenableaneedforastatisticalproducttobeevaluatedwithrespecttoitsscope,detailorcost?

• Towhatextentarequalityguidelinesdocumentedbytheagency?• Isstatisticalconfidentialityguaranteed,andifso,underwhatauthority?• Towhatextent,andhowquickly,areanyidentifiederrorsinpublishedstatisticscorrectedandpublicized?

22

Australian Bureau of Statistics Recommendation

These questions can be tailored for companies and their departments.

Change Culture or Get Disrupted

23

“Culture eats strategy for breakfast, technology for lunch, and products for dinner, and soon thereafter everything else too.” – Business Management Guru Peter Drucker

24

This culture change is critical to winning with data. No amount of strategizing will work otherwise. However, most companies are in

denial that there is a culture problem.

25

“Succeeding with data isn’t just a matter of putting Hadoop in your machine room, or hiring some physicists with crazy math skills. It requires you to develop a data culture that involves people throughout the organization.” - DJ Patil, Chief Data Scientist of the U.S.

Winning with data isn’t about Hadoop or new technology. It requires you to develop a data culture that involves everyone.

Let’s Define Culture First

Aculture isawayoflifeofagroupofpeople--thebehaviors,beliefs,values,andsymbolsthattheyaccept,generallywithoutthinkingaboutthem,andthatarepassedalongbycommunicationandimitationfromonegenerationtothenext.Culture issymboliccommunication.

26

Here Are 11 Characteristics of a Data Culture1. MissionAlignment:Data’sroleinthecompany’soverallmissionandgoalsisclearlyarticulated.Openlydiscussing

strategiesandinnovationgoalsprovidesemployeeswithaclearviewofdata’sroleinthecompany’soverallmissionandreinforcestheirconnectiontothelargerorganization.

2. DataQualitySavvy:Managementandstaffthataredataqualitysavvy– understandallthefoundationalelementsofdataqualityandwhydataiscriticalforsuccess

3. DataQualityProcesses:Definequalityrequirements,measurequality,andproactivelyaddressqualityissues4. Behaviors:Everyonemakesevidence-baseddecisions(notbasedongut)5. RightQuestions:Leadersandstaffareempoweredtoasktherightquestionssuchas– whatisthesystemofrecordfor

data?,what’sbeendonetoit?,canItrustit?,whoisaccountableforspecificdata?etc.6. InformationSupplyChain:Departmentalsilosofinformationarethenemesisofthrivingdatacultures.Topromotethe

viewofdataasaflexibleassetthat’susablebymultipledepartments,organizationsneedtoeducateemployeesonhowthedatatheyusedailyripplesthroughotherpartsoftheorganization.Employeesneedtoseethebigpicture.

7. RewardsandRecognition:Datasuccessesaresharedandindividualsandteamsresponsibleforthemarerewardedandrecognized

8. RightIncentivesandAlignment:Cross-functionalsolutionteamsarecompletelyalignedongoalsandincentivesbetweenIT,Data,andBusinessstaff

9. DataSharing:Thereissharingofdataandinformationbetweendepartmentsandtotaltransparency– nodatahoarding.Athrivingdataculturedependsonanenvironmentinwhicheveryonecanshareinformationwithoutbeingperceivedasnegative.

10. KPITransparency:Availabilityanduseofkeydatametricsandmeasuresviacomprehensivedashboard– dataquality,dataissuemanagement,datagovernance,datasecurityandprivacy,datalineage,etc

11. RobustDataPlatform:Arobustdataplatformhasbeenbuiltanditsupportsthetypesofanalyticsrequiredtomakedecisions,managerisk,andinnovate

27

The Culture Change Framework

OrganizationalCultureChange

BusinessCaseForChange

Creatingtheclimate forchange

Implementingand

SustainingChange

Engagingandenablingthe

organization

1. Establishingasenseofurgency2. Creatingtheguidingcoalition3. Developmentofachangevision

1. Communicatingthevisionforbuy-in2. Empoweringbroad-basedaction3. Generatingshorttermwins

1. Neverlettingup2. Incorporating

changesintotheculture

1. Establishingtheneedforchange2. Tyingculturechangeinitiativetobusinesspriorities3. Articulatingthevisionandtangibleresults

Team-Building ProcessesTeam building is a development process where the course is divided

into 4 phases plus a resolution phase.

Performan

ce

Courseoftime

Low

High

Orientationphase(forming)1

Growthphase(performing)4

Cooperationphase(norming)3Confrontationphase

(storming)2

Resolutionphase(adjourning)5

Call To Action

30

Every great dream begins with a dreamer. Always remember you have within you the strength, the patience and the passionto reach for the stars to change the worldHarriet Tubman, Abolitionist, Humanitarian & Spy (1822 – 1913)

“Mybiggestfearisnotcrashingonabike…

It’ssittinginachairat90andsaying,

‘IwishIhaddonemore’.”GraemeObree

The Climb– It’s Tough But Helps You Win

1. Companyleadershipneedstoelevatedataqualityasatoppriorityandtieittotangiblecustomer,product,andemployeebenefits.

2. Dataproducersandconsumersmustagreeanamutualcontractanddeliveronit.Anindependentauditregimechecksadherencetocontractterms.

3. Developaroadmapforculturechangeandimplementit.4. Investintraining.5. Buildworldclassdataqualityprocessexecutioncapabilities.

Appendix

34

17 Articles I’ve Authored On Data Quality

• 5Reasons WhyMoreCompaniesDon'tHaveDataQualityProcessesInPlace?• DataQualityisJob1andHere'sWhy?• SeriousImplicationsoftheDarkSideofBigData• Whatisthisthingcalled"DataQuality"?• HolisticDataQuality(HDQ)- ANewParadigmInEnterpriseDataQualityManagement• TheHolisticDataQualityFramework- Version1.0• HighQualityData+Analytics=DeepInsights• Here'sWhyYourDataDoesn'tReconcile?• 5DataQualityBestPractices• ShouldUsersSwitchFromOfficeProductivityToolsToCommercialDataQualityTools• 7RootCausesFortheHighCostofBadData• BadDataIsCostingtheU.S.AtLeast6%ofItsGDP• 6CoreComponentsofaDataQualityProgram• 3ActionsCanSaveYourOrganizationMillions• OrganizationCanSaveMillionsByApplyingThisDataRule• 4IngredientsForProducingTrustworthyBigData• UntrustworthyDataIsSpawning"ShadowITandDataNinjas"

35

Here’s What’s Required To Win1. Companyleadershipneedstoelevatedataqualityasatoppriority

• Everyoneinthecompanyshouldbecomedataqualityconsciousandincorporatequalityintoeveryprocessstep• Benchmarkthecompany’sdataqualitypracticesagainstbestinclasscompaniestoidentifygaps• Assessmaturityofdataqualitycapabilities(i.e.,people,process,technologyanddata)toidentifyareasofimprovement

2. DataProducersandConsumersMustAgreeOnAMutualContract• Contractprovidestransparencyintodataset,itscontents,anditsqualityfromproducers• Producersmustcertifydatasets,basedonpre-definedconsumerexpectations• Anobjectivethirdpartyshouldauditthecontractsandtheartifactstoensurethatproducersandconsumersareworkingingood faith

3. Developaroadmapforculturechange• Galvanizeleadersandassociatestobecome“dataqualityfocused”andtoinvestindataqualitymanagement• Incrementallybuilddataqualityprofilingandremediationcapabilitiesinanopportunisticmanner(withfocusonbusinessresults)• Influencepeerstobuildacoalitionforchangetoimprovequalityofdata

4. Investintraining• Trainleadersandassociatesinchangemanagement• Trainleadersandassociatesonnewframeworks,technologies,executionstrategiesfordataquality• Educateandraiseawarenessaboutnewdataqualitycapabilitiesamongstpeersandleaders

5. BuildWorldClassExecution• Pickspecificareaswithinyourdepartmentwhereyoucanshowtangibledirectimprovementtothebottomlinebyapplyingthenew

executionmodel

36Becomingdataconsciousandimplementingqualitycontrolrequiresculturechange,

investments,andalongtermview.