Considerations for Streamlining YourBusiness Intelligence Implementation
March 2008
© 2008 Endasol, Inc. All Rights Reserved.
© 2008 Endasol, Inc. All Rights Reserved.
Brief Introduction - Endasol (www.endasol.com)
Focus AreasData Warehouses/ BI Efforts
Lead Roles in PJM Enterprise-Wide Data Warehouse prior to forming EndasolCalled back to PJM to help develop a separate Market Monitoring Data MartJust underway with helping to identify business requirements and early design for a major data mart at an ISO.Business Analysts for a couple different energy companies’ major data consolidation & reporting projectsDeveloped Comprehensive Solution for Wholesale Power P&L + Other KPIs
Systems Integration – ETRM, Scheduling, Settlements, Generator Management, Fuel InventoryISO Operations & Interfaces - PJM, MISO, NYISO, ISO-NE
Developed Streamlined Interfaces to many of the PJM eSuite ApplicationsDirect Experience in estimating Major Settlement Line Items
Business Process Development/ Refinement
© 2008 Endasol, Inc. All Rights Reserved.
Streamlining BI Implementation - Topics
Components of Business Intelligence
Methodologies for Streamlining a BI Implementation
Relationship to Business Process Management
© 2008 Endasol, Inc. All Rights Reserved.
Components of BI Systems
© 2008 Endasol, Inc. All Rights Reserved.
BI System Integration & Information Layers
Data WarehouseData movement from Sources to Target Tables
Usually scheduled in batched ETL processesStaging Tables or Operational Data Store often an intermediate step
Data Model promotes business user reporting & querying – DimensionalData Propagation
Data movement among SystemsUsually in Real Time or near Real TimeUse of Enterprise Application Integration (EAI) tools to publish & subscribe to data
Data Warehouse can be a SubscriberData Federation
No data movementPre-established Views are often constructed to relate data across systems
More suitable for highly decentralized organizationsA sensible approach may involve elements of each method
© 2008 Endasol, Inc. All Rights Reserved.
BI System Integration Layer
Pre-established ReportsRoutine reports designed & developed in advance – often scheduled Typically ‘locked down’ and controlled by a couple of individuals - often ITMay build intermediate tables, views and/or calculated fields to feed reports
Self Service Reports and QueriesBusiness users should be able to do reporting & querying without the need for IT
Business users do not always know exactly what they want until they start using the data in ways not previously possibleRequires a sound foundation (typically dimensional data model), data dictionary or other metadata, and training
OLAP CubesOLAP provides interactive analysis and reporting using multidimensional views of data
Users can slice, dice, drill-down and roll-up data – more than Excel pivotsCubes are often pre-built and refreshed right after batch ETL processes so that they are immediately available to users
© 2008 Endasol, Inc. All Rights Reserved.
BI System Intelligence Layer
The value of the BI System is often not realized until deploying this Layer
Advanced Visualization, including Dashboards & ScorecardsEnable rapid assessment of Business Metrics or Key Performance IndicatorsDashboards display information in tables, charts, gauges, and other indicatorsScorecards display metrics against pre-established thresholds grouped into red, yellow, green or numerical weightingsUltimately, should have the capability to drill down into information feeding the dashboard and scorecard items
© 2008 Endasol, Inc. All Rights Reserved.
BI System Intelligence Layer
Advanced AnalyticsExploratory Data Mining – seeking new insights through examining the information with statistical analysis or other techniquesPredictive Analytics – formulating rule-based models to predict what will happen under various scenarios with high degree of confidence
Many BI efforts are abandoned before reaching this LayerWhy? Because implementation of the other layers have been overly costly with mediocre results
Predictive Analytics; Extending the Value of Your Data Warehousing Investment, 2007 TDWI Best Practices Report, Wayne W. Eckerson
© 2008 Endasol, Inc. All Rights Reserved.
Employ methods that get to the higher value BI layers sooner:
Establish Fundamentals Early
Consider using a Packaged Solution or Starter Model
Consider Agile Development Approach
Consider Hybrid Approach
Considerations for the Interactive Layer
Considerations for the Intelligence Layer
Streamlining BI Implementation
© 2008 Endasol, Inc. All Rights Reserved.
Implement a Data Governance Program UpfrontAssign Data Stewards from the Business responsible for quality and use of data
Establish Monitoring and Methods for Correcting Issues
Consider a Master Data Management ProgramManaging your Reference Data, such as customer, generatorsEnsuring consistency and dynamic mapping among systemsSome examples of the need:
How many systems or databases house customers (individuals or organizations) and how many different identifications, abbreviations, or alias’s are used?How are you tracking who owns whom? Frequently changes - if not careful, you may have excess credit exposure to entities or entities may be exercising unfair market power.
Various technologies are availableEnsuring consistency among systems regarding Master Data will streamline BI implementation, but not absolutely needed since the data warehouse will ultimately solve this issue for users of the data warehouse
Establish Fundamentals Early for Any Phase
© 2008 Endasol, Inc. All Rights Reserved.
Set realistic expectations from both business & IT perspectivesDefine precise & measurable objectivesEnsure that the BI system will be scalable to accommodate other corporate initiatives in the futureMore importantly, align the project with overall corporate BI objectives
Minimize scope creepWell-defined business requirementsEstablish Change Control policies upfront – know how changes will be processed and who needs to approve these
If you don’t have the expertise in-house, then hire professionalsEnsure they have experience in the phase you are embarking onEnsure those on the team have experience with your businessBe involved – you still need to ‘own’ the program & business must be involved
BI systems are not static – you will need skills to advance & maintain your system
If not prepared to adopt these & other fundamentals, then don’t start!
Establish Fundamentals Early for Any Phase
© 2008 Endasol, Inc. All Rights Reserved.
Packaged BI Solutions: 3rd party products fulfilling a slice of your BI needs
A Packaged BI Solution should:Provide the business with a clear definition of the end results – what you will get is visible from the start!Integrate and merge data across all appropriate systemsEnable easy access to underlying data and the information it providesServe as a starting point for other analysis and reportsBuild on an existing data warehouse as appropriate
Use of Packaged BI Solutions & Starter Models
© 2008 Endasol, Inc. All Rights Reserved.
A Packaged BI Solution is typically not:A front line application, such as an Energy Trade & Risk Management System (ETRM), or a Customer Information System (CIS) - too many try to elevate these transactional systems to be an enterprise data mart for reports & analysis
Are there Packaged BI Solutions available in our industry?For wholesale power, including financial transactions, we have our PREMIRTM SolutionThere may be solutions available for the Retail, and Transmission/Distribution SectorsConsulting Companies focused in your business may offer at least Starter Models
Use of Packaged BI Solutions & Starter Models
© 2008 Endasol, Inc. All Rights Reserved.
Endasol’s Flow Diagram
Our present focus is the Wholesale Power & Financial Transactions Data Models and Reporting
premir SolutionsTMpremir SolutionsTM
© 2008 Endasol, Inc. All Rights Reserved.
Advantages:Requirements/Vision already defined – you see what the end result looks likeData Model provided and can be built uponKey Reports may be provided and others can be addedKPI’s & Dashboard may be provided and can be expandedIn any case, they can provide a head start enabling quicker deployment
Potential Pitfalls:Don’t expect a packaged solution or even starter model to meet all of your objectives – expect that some customization will be neededCompatibility with existing data warehouse, data marts must be consideredCompatibility of technology used – database, reporting tools – must be consideredCapability to expand the solution to other areas of the enterprise must exist
Conclusion: A packaged solution or starter model can significantly expedite your BI implementation, but do your homework first
Use of Packaged BI Solutions & Starter Models
© 2008 Endasol, Inc. All Rights Reserved.
Waterfall Methodology – Traditional ApproachFrom Wikipedia: “The waterfall model is a sequential software development model (a process for the creation of software) in which development is seen as flowing steadily downwards (like a waterfall) through the phases of requirements analysis, design, implementation, testing (validation), integration, and maintenance.”Often characterized by lengthy requirements, design and signoff before any development is allowed to begin.
Agile Methodology – Currently gaining acceptanceFrom Ralph Kimball: “…In general, it (Agile) refers to a group of methodologies, including Extreme Programming, SCRUM, Adaptive Software Development and others, which share a common focus on iterative development and minimizing risk by delivering new functionality in short timeframes, often measured in weeks.”Less structured with emphasis on deliverables, not documents & signoffs
Consider Agile Development for BI - Background
© 2008 Endasol, Inc. All Rights Reserved.
Consider Agile Development for BI
Agile Development is gaining interest - Agile is now included in training classes and conferencesScrum is one structured methodology for Agile Development
Product/Project broken into deliverables, called ‘Sprints’ of 2 to 4 weeks.Sprints established to bring priority, but achievable results within time frame.Sprints typically locked down with high resistance to changing scope once started.Product Owner - The Product Owner represents the voice of the client. He/She ensures that the Scrum Team works with the right things from a business perspective.ScrumMaster (or Facilitator) - Scrum is facilitated by a ScrumMaster, whose primary job is to remove impediments to the ability of the team to deliver the sprint goal. The ScrumMaster is not the leader of the team (as they are self-organizing) but acts as a buffer between the team and any distracting influences. Empowered teams without traditional Project Management.Scrum Team - A small team of 2+ people with cross-functional skills to do the actual work (designer, developer etc.).Daily meetings among players – IT & Business.Development proceeds immediately and documentation often lags.
© 2008 Endasol, Inc. All Rights Reserved.
Consider Agile Development for BI Implementation
Biggest Issue with many DW/BI projectsChanges during the project!
Source system changesRequirementsData quality issues – some sources are just not very good.Change in business sponsor / business ownerMarket changes
ISO rule changesSubscription services
Business Priorities
© 2008 Endasol, Inc. All Rights Reserved.
Consider Agile Development for BI Implementation
Advantages of Agile Development Design at the beginning of each sprint - Learn from each sprint Phased / iterative approach – Useable tools delivered quickerDetermine next sprint based on priorities.Working closely with the business users during data cleansing and development of reports, cubes and dashboards
Increase user satisfaction and ‘ownership’Decrease ‘do-overs’
Building what the business needs overrides building just to meet documentation
© 2008 Endasol, Inc. All Rights Reserved.
Consider Agile Development for BI Implementation
Disadvantages of Agile DevelopmentBudgeting difficulties due to minimal upfront analysis & design
Hardware needsStaffing – how long?When is the project considered finished?
How does the present sprint fit into the solution?Some rework may be neededHave a risk of delivering isolated silos of data which meet individual releases, but don’t integrate well with each other
New to the organization and not well understood by IT and Business. Users are confused about their role.Users may not have sufficient time.Minimize risk by using company standards & following corporate SDLC process
Documentation – Agile methodology minimizes documentation. What about support and future enhancements?
© 2008 Endasol, Inc. All Rights Reserved.
Hybrid Approach for BI Implementation
Optimum Approach may be a Hybrid between Waterfall & AgileTake time upfront to develop overall BI plan/ roadmap
Spend some time on how the overall BI solution fits into the enterprise pictureIdentify subject areas for potential phasesDevelop framework of data modelSet standards Select hardware / software
Break into focused phasesPhase 1 may be 4 – 6 months due to infrastructure, overall requirements, design & documentation Focus mainly on current phase subject area but know how it fits with the other areas.Other phases may decrease in length as more and more subject areas are added.
Take each phase all the way through user reports, OLAP cubes, dashboard & scorecardInclude support with the delivery of each phase
© 2008 Endasol, Inc. All Rights Reserved.
Hybrid Approach for BI Implementation
DetailedRequirements
Establish Project•BI System Objectives•Business Requirements•Prioritize Phases •Project Plan
High Level Design•Subject Area Data•Subject Area Relationships•High Level Data Model•Set Standards
Infrastructure•Hardware•Software/ Licenses•Set Up Environments
DetailedDesign
ETL &Test
UserReports
OLAPCubes
Dashboard/ Scorecard
Phase1
Phase 2
Phase3
Phase4
Repeatfor
eachPhase
EachPhase
may getshorter
© 2008 Endasol, Inc. All Rights Reserved.
Choosing Reporting and Querying ToolsBusiness Ease of Use
Deploying a mixture of tools
What about Excel?MS Excel will continue to be the tool of choice for many business usersEven with well managed data warehouses, end users still resort to ExcelBut this may not be bad as long as the source is the data warehouse and not direct pulls from source systems
How important is OLAP?
Streamlining Interactive Layer
© 2008 Endasol, Inc. All Rights Reserved.
Give Business Early Access to DataStandard reports should be pre-established using the corporate reporting tool & ‘Locked Down’Underlying structure/ data model should be such that business users can easily understandAllow them to develop their own reports and analysis - Stress using corporate reporting tools, but be flexible as long as they are sourcing from the data mart
Streamlining Interactive Layer
© 2008 Endasol, Inc. All Rights Reserved.
Restrict Key Performance Indicators (KPIs) to a small number (6 –10) specific to the organization’s goalsDevelop different set of KPIs for each business unit, sub-unit
The business unit’s KPIs should relate to corporate KPIsThe business unit’s KPIs should also include what’s important to bonuses
Most effective use is often a mixture of financial & non-financial KPIs as in the Kaplan and Norton Balanced Scorecard approachKPIs should follow the SMART test:
Specific, Measurable, Achievable (Actionable), Realistic, Timely
Once implemented, continue to refine KPIsRevise as corporate and business unit objectives changeMonitor who is accessing and how often –retire KPIs not being usedEnsure individuals can easily access KPIs and understand what they are viewing
Streamlining Intelligence Layer - KPIs
© 2008 Endasol, Inc. All Rights Reserved.
Choosing TechnologyNumerous products available with nice dashboard & scorecard features
Some are limited to specific technologies, such as using MS Analysis Services (cubes) to drive dashboard components
Most effective is ability to drill down – start with the picture & drill to underlying information
Don’t need to go overboard on dashboard/ scorecard technologyIf can’t justify expense upfront, then consider publishing output of certain reports & OLAP such as graphs or charts in a web-portal – this can be automatedOnce base functionality is working, can always buy advanced technology later
Streamlining Intelligence Layer – Visualizing
© 2008 Endasol, Inc. All Rights Reserved.
Implementing Dashboards & Scorecards
Keep it simple – don’t clutter the viewing screen
Consider allowing viewers to construct their own ‘myBI’ pageSome items may be required viewing such as important metricsOthers may be selected from a list of available dashboard reports, graphs, charts
Continue to monitor usage and effectiveness & continue to refine
Streamlining Intelligence Layer – Visualizing
© 2008 Endasol, Inc. All Rights Reserved.
Determine Data Mining or Advanced Analytical Methodology Early
Exploratory – looking for new insights and/ orPredictive – developing models to predict behavior/ market opportunitiesDifferent methodologies are available for each focusMethodology may require very specific data model or structure
Data model used for advanced analytics may conflict with that optimal for reporting and user-queriesIf you know this upfront, you may be able to build a hybrid model that can serve both purposes, orYou may be able to implement a model structure that can be more easily transformed into that needed for the advanced analytic application
Streamlining Intelligence Layer – Data Mining
© 2008 Endasol, Inc. All Rights Reserved.
Ultimately, your BI System should be aligned to corporate objectives and provide monitoring & feedback for continued alignment
Corporate KPIs should trickle down to corresponding KPIs pertinent to individual Business Units
Business Unit KPIs should link to corporate & adjusted as corporate objectives change
The above is part of the broader initiative referred to as Business Performance Management (BPM), Corporate Performance Management (CPM), or Enterprise Performance Management (EPM)
There are advancement in software tools which support implementation & monitoring
Align Your BI System with Corporate Objectives
© 2008 Endasol, Inc. All Rights Reserved.
Streamlining BI Implementation - Summary
Align your BI System with Corporate Objectives
Value of your BI System is often not realized until deployment of Visualization and Advanced Analytics
Dashboards & Scorecards to track KPIs & other important measures
Drill-down into information driving displays is an important capability
Data Mining for discovering new insights
Predictive Analytics to gain edge in market
© 2008 Endasol, Inc. All Rights Reserved.
Streamlining BI Implementation - Summary
Objective is to get to these value BI Layers as soon as possible:Use Packaged BI Solutions or Starter Models if available and if applicableUse a Hybrid Approach of Agile & traditional Waterfall Development
Do some detailed requirements & design upfrontBreak Project into short phases of 1 – 6 months covering selected subject areasEach Phase should include all BI Layers - Data Model, User Reports, Cubes, Dashboard & Scorecard items
Technology – decide what’s important to you from a budget and capability perspective, be flexible and don’t get stuck in analysis paralysis.