![Page 1: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/1.jpg)
Lean Data Architecture:Minimize Investment,
Maximize Value
Brandon [email protected]@BrandonByars
Manvir [email protected]@ManvirGrewal
![Page 2: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/2.jpg)
Brave New World
Every 2 days we create as much informationas we did from the dawn of civilization to 2003
- Eric Schmidt, 2011
Big Data• Volume• Velocity• Variety
NoSQL•Schemaless•Non-relational
![Page 3: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/3.jpg)
Future, Backwards
![Page 4: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/4.jpg)
Data Platform
![Page 5: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/5.jpg)
Support variety of analytics/users
![Page 6: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/6.jpg)
Ubiquitous access
Logicwolf.com
![Page 7: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/7.jpg)
Evolves as business changes
![Page 8: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/8.jpg)
Sound good…So what's stopping you?
![Page 9: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/9.jpg)
Companies lose $8.2 Million annually due to bad data – Gartner
![Page 10: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/10.jpg)
Study estimated total cost to US economy of data quality problems at over US$600 billion per annum - Wayne Eckerson
![Page 11: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/11.jpg)
SpreadMarts
![Page 12: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/12.jpg)
![Page 13: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/13.jpg)
Didn’t test well enough…
![Page 14: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/14.jpg)
Upfront and Inflexible design
Vk.com
![Page 15: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/15.jpg)
Bring everything, we may need it
![Page 16: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/16.jpg)
![Page 17: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/17.jpg)
Manual Testing is good for…
![Page 18: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/18.jpg)
Not good for data or software
![Page 19: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/19.jpg)
Spreadsheet comparisons…over and over!
![Page 20: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/20.jpg)
Where are your bottlenecks?
![Page 21: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/21.jpg)
![Page 22: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/22.jpg)
The Data Focus Problem
"Data warehousing and BI is one of the largest expenses in CIO budgets, and to-date has had limited value to the business."
- Jeff Smith, CEO Shared Services, Suncorp
Gartner estimates that between 70-80% of BI projects are failures
![Page 23: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/23.jpg)
How do we get to the future?
Wallchan.com
![Page 24: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/24.jpg)
App Analysis Modeling DBA ETL Testing Report
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks4 weeks
2 weeks
Customer
![Page 25: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/25.jpg)
Value Streams
App Analysis Modeling
![Page 26: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/26.jpg)
Systems Thinking
How long would it take you to add a single column to a report?
From Idea to Prod…
![Page 27: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/27.jpg)
Muda
![Page 28: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/28.jpg)
Eight Wastes
Logismarket.de
![Page 29: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/29.jpg)
BI/DW: Status Quo
Data
![Page 30: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/30.jpg)
Lean BI: Value First
value
Cost & Time
100%
100%
![Page 31: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/31.jpg)
MVP
Sometimes we accidentally build something that nobody wants, in which case it doesn’t matter if we do it on time and on budget
Prototype reportDashboard populated with static dataRudimentary scoring model
![Page 32: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/32.jpg)
Reduce Batch Size
Dev2ops.org
![Page 33: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/33.jpg)
Reduce WIP
![Page 34: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/34.jpg)
Work In Progress
L = λW
![Page 35: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/35.jpg)
Simplify Architecture – Message Based
ReportsListenerBus
DB
DB
DB DB
![Page 36: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/36.jpg)
Automated Testing
![Page 37: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/37.jpg)
How do we test?
Test data set based on profilingVolume is not necessaryVariety is necessaryVariability/Complexity is necessaryVelocity is necessary
System testingPlay transactions from sources in UAT/pre-prod environment
![Page 38: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/38.jpg)
Developer Sandboxes
![Page 39: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/39.jpg)
Automated Build
![Page 40: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/40.jpg)
Continuous Integration
![Page 41: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/41.jpg)
References
![Page 42: Lean Data Architecture: Minimize Investment, Maximize Value Brandon Byars bbyars@thoughtworks.com @BrandonByars Manvir Grewal mgrewal@thoughtworks.com](https://reader036.vdocument.in/reader036/viewer/2022062301/56649eb75503460f94bc1645/html5/thumbnails/42.jpg)
Data Explosion is coming!