life in hell: the experience of successful bi managers
DESCRIPTION
Life in Hell: The Experience of Successful BI Managers - retitled from the boring original. Growing business intelligence from a departmental deployment or pilot project to an enterprise scale is not as simple as buying more licenses. Increasing the scope of BI means taking into account a broader set of information needs in the organization, increasing the size and variety of data needed in a warehouse or mart. Broader use implies more varied delivery needs, as well as scaling up the user count. Increasing the scope of use also raises the importance of managing day to day operations and reliability. This webcast will discuss factors to consider when increasing the scope of a BI program, and what requirements that puts on components of the technology infrastructure. Different axes of enterprise scale How information delivery requirements change as you expand BI The impacts of growth on operations and administration Slides from a webcast for TDWI. You can listen to the full webcast and see the Jaspersoft/Talend presentations here: http://ow.ly/3lkHjTRANSCRIPT
1
What's Required to Grow an Enterprise BI Deployment?
December 7, 2010Mark Madsenwww.ThirdNature.net
Maturation of BI
A healthy BI program gets more complex and harder to manage over time.
Growth from the initial installation to broader organizational use has many aspects:▪ Number of users
▪ Multiplying uses
▪ More information
Growth requires adapting processes and technology.
Maturation: Initial Build, the Early Stage
Projects
ROI Minimum ROI hurdle for initial project to be built, infrastructure created
Future work planned for
Done
Consultants leave here
Copyright Third Nature, Inc.
The Origin of BI Backlog: Next Phase
Projects
ROI
Minimum ROI hurdle is lower for subsequent work
Done
Fewer resources, so work takes slightly longer to complete, but not so long as initial build
Copyright Third Nature, Inc.
The Long Tail of BI
Projects
ROI
This is what happens to successful data warehouses
Done
“Oh crap”
Copyright Third Nature, Inc.
Prioritizing the Long Tail of BI
Projects
ROI
Financial priorities, business priorities, steering committees, budget limits, time-boxing…(Guess which things get done)
Copyright Third Nature, Inc.
Prioritizing the Long Tail of BI
Projects
ROI
Executive pet projects
Financial priorities, business priorities, steering committees, budget limits, time-boxing…(Guess which things get done)
Copyright Third Nature, Inc.
The Long Tail of BI: Why We Have Spreadmarts
Projects
ROI
The Kingdom of ExcelLow hanging fruit
*sigh*
Mismanage this process and you have a legacy system everyone complains about
Copyright Third Nature, Inc.
Requirements, data sources
ETL / DI
Warehouse / Mart
BI / Analytics server
Clients
Development Process Designed to Minimize Later Change
Metadata stored here, almost but not quite the same each time
The common MD repo was supposed to fix this
This process is fine for the initial build.
Three months later, not so much.
Two Things People Don’t Want
Data integration and BI projects that take months to deliver for business needs that may be one‐time or done in weeks.
Least‐common denominator financial and transaction data with contextual information and details stripped away in the name of speed.
Slide 10
The Process is Not a Waterfall, It’s an Ongoing Cycle
The BI layer is the starting point for users, not the end point.
As people adopt new information, needs alter, driving change.
Your processes switch from “build” to “keep it running”.
Requirements, data sources
ETL / DI
Warehouse / Mart
BI / Analytics server
Clients
Slide 11Copyright Third Nature, Inc.
Growth: Increased UseIf you’re successful, users become more proficient and BI use increases.
Effects are:▪ Performance problems
▪ Capacity problems
▪ Shrinking data load windows and response time requirements
This presages more user growth.
Raises problems of scaling not only performance, but also managing accounts, data security, licenses.
Aspects of Growth: More Users
Growth: Increased Variety of Uses
Once people master the basics, diversifying demands require new tools, more complex analysis and models.
Warning about software vendors:
The Swiss knew when to stop. Vendors often don’t.
BI and DI vendor response has been to add features to the tools to meet all the different use cases.
Different uses can drive conflicting tool requirements.
Growth: Data
Data is the item everyone focuses on when talking about growth and BI.
The primary impact is on the database, both getting data out and getting data in.
Data volume is the easiest problem to address (in most common BI / DW situations).
Other aspects of data growth are harder to address
Variety:1.More sources▪ More system types
▪ APIs and other oddities
2.More types▪ The usual suspects
▪ And more
3.Uses that require more complex data models or transformations.▪ Like data mining
▪ And sandboxes
Growth often drives the need for lower latency12x5 moves to 24x7 operation, driving SLAs, capacity planning, failover and disaster recovery.
Methodology, organization and technology are related. Speeding up one won’t always speed up the others.
18
New Data Integration Methods and Tools Required
The initially designed ETL may not handle the varied data latencies, or deliver the performance to meet smaller batch windows.
Daily+
Single batch
Frequent batch
Continuous load
Streaming
Immediate
Mini‐batch
Copyright Third Nature, Inc.
Development, Maintenance & Operations
Real time decisions on low latency data mean data quality plays a larger role, and it’s harder to address.
Warehouse availability becomes much more important to the business, and it isn’t just the database – it’s everything.
Performance and meeting strict BI SLAs will rise in importance since you are now tied in to business operations.
Slide 20
Administration
As BI grows in importance within the organization this becomes a focal point.▪ Any problem is magnified due to the broader scope.
▪ There are more products.
▪ There are more inter‐tool dependencies and problems are more distributed.
▪ The least‐emphasized set of features in most BI tools.
▪ And most DI tools.
Overall Effect of Growth
Complexity, which leads to• Performance problems
• Reliability problems
• Maintenance problems
• Difficulty adapting to change
Usually we forego features useful in the long term during our product evaluations in favor of features important for initial delivery.
About the Presenters
Mark Madsen is president of Third Nature, a technology research and consulting firm focused on analytics, business intelligence and data management. Mark is an award‐winning author, architect and CTO whose work has been featured in numerous industry publications. He is an international speaker, a contributing editor at Intelligent Enterprise, and manages the open source channel at the Business Intelligence Network. For more information or to contact Mark, visit http://ThirdNature.net.