An empirical investigation of metadata issues in Business Intelligence

environment for Higher Education Institutions

Yuriy Verbitskiy

Principal supervisor: William YeohAssociate supervisor: Andy Koronios

Minor Thesis final presentation

2

Outline

• Introduction: main principles of BI, BI environment, motivation and research question

• Literature review: BI issues, requirements and similar research

• Action research design• Reasons for providing metadata in BI and

requirements for metadata solution• Metadata solution: architecture, Metadata Framework

and metadata prototype• Metadata prototype implementation• Conclusions

3

Introduction – main principles of BI

• Business Intelligence (BI) is on the top of priority list for CIO worldwide during the last 3 years (Gartner)

• BI - is a set of concepts, methods, and technologies

• BI has a number of issues, such as:• Understanding of BI environment• Understanding of data it delivers

• Making decisions based on the results of BI tools is the biggest challenge for users (Lawton 2006)

4

Introduction – BI environment

SQL

Excel

XMLXML

DB2

… Data Warehouse

ETLETL

Data marts

OLAP

Reports

Business applications

Sales amounts

Metadata repository

Business rules

Business rules

Dashboards

5

Introduction – motivation and research question

• Australian universities use BI technology for different tasks

• To apply BI technology successfully, Australian universities require a metadata solution.

• Issues in the metadata implementation:• No standard approach for developing the metadata• Complexity of the metadata implementation

• This research investigates how to improve the metadata implementation for the BI environment in Higher Education Institutions.

6

Literature review – BI issues

• There are “three enterprise standards that are prerequisites to delivering a consistent single version of the truth” (Beyer 2007), which are: terminology, calculation and methodologies.

• “People and organization” is the most significant obstacle for supporting BI (Burton, Popkin et al. 2007).

• Staff members, who work on different layers in BI environment, tend to speak a different language (Chisholm 2008)

7

Literature review – requirements

• Indirect usage (Inmon, O'Neil et al. 2008)• Centralized metadata repository (Paolo Missier, Pinar

Alper et al. 2007; Inmon, O'Neil et al. 2008)• Interoperability or (API) for access by other software

(Vaduva and Dittrich 2001; Ward 2007)• Interchangeable metadata format

(Shankaranarayanan and Even 2006)• Browse, search, filters, facets (Vaduva and Dittrich

2001; Ward 2007; Foulonneau and Riley 2008)

8

Literature review – similar research

• A comprehensive repository model for managing the data warehouse metadata (Stöhr, Müller et al. 1999)

• A software architecture for metadata management that was developed for the data warehouse environment (Auth, Maur et al. 2002)

• A multi-dimensional metadata framework for the enterprise business intelligence (Stephens 2004)

• Metadata version and configuration management is extensively discussed in Friedrich (2005)

9

Action research design

10

Reasons for providing metadata in BI

• To provide consistency for descriptions and definitions of the data in BI environment

• To provide an overall enterprise view• To solve the problem of misinterpretation of some

terms which have different meanings for staff with different roles

• To provide translation between technical and business terms

11

Requirements for metadata implementationArea Requirement       Priority

Presentation of metadata        

  Layered presentation of metadata     MEDIUM

  Providing the names of contact person, email   HIGH

  Browsing, Searching, Facets, Key words, Filters   HIGH

             

Metadata repository          

  Easy customization of metadata structure in the future HIGH

  Hierarchic metadata classification     HIGH

  Metadata structure is shown in metadata model to help users HIGH

Refreshing of metadata from various sources on a regular basis HIGH

Import/Export functionality with Excel     HIGH

             

Metadata infrastructure          

Accessibility from multiple places, uniform access mechanism MEDIUM

Integration with existing BI environment, context-sensitivity HIGH

  Interchangeable metadata format     MEDIUM

  API for access by other software applications   MEDIUM

 

Metadata management          

  Easy to support and change     HIGH

  Metadata stewardship       HIGH

  Access control       HIGH

Metadata change technique     HIGH

  Metadata version management strategy   LOW

  Notification mechanism       LOW

  Metadata quality       HIGH

12

Metadata solution – architecture

ASC

BAC

CAS

# Name Date

1 Fi sh 01/03/08

2 Bread 05/04/08

3 M eat 21 /03/08

Metadata repository

SQL Server 2005

Metadata interface

ASP.NET 2.0

BI environment

Cognos 8.4

ADO.NETWeb Services

Metadata Framework

Other Metadata sources

ASP Page

Excel files

Cognos JavaScript

pages

13

Metadata solution – Metadata Framework and prototype

Area Requirement         Priority Framework Prototype

Presentation of metadata        

  Layered presentation of metadata     MEDIUM Not discussed Not implemented

  Providing the names of contact person, email   HIGH Discussed Implemented

  Browsing, Searching, Facets, Key words, Filters HIGH    

               

Metadata repository            

  Easy customization of metadata structure in the future HIGH    

  Hierarchic metadata classification     HIGH    

  Metadata structure is shown in metadata model to help users HIGH    

Refreshing of metadata from various sources on a regular basis HIGH    

Import/Export functionality with Excel   HIGH    

               

Metadata infrastructure          

Accessibility from multiple places, uniform access mechanism MEDIUM    

Integration with existing BI environment, context-sensitivity HIGH   Interchangeable metadata format     MEDIUM    

  API for access by other software applications   MEDIUM    

  Metadata management          

  Easy to support and change     HIGH    

  Metadata stewardship       HIGH    

  Access control       HIGH    

Metadata change technique     HIGH    

  Metadata version management strategy   LOW    

  Notification mechanism     LOW    

  Metadata quality       HIGH    

14

Metadata prototype implementation

• During the practical implementation two issues appeared:• metadata implementation solution very much depends on

the functionalities of the BI environment• a need to understand how the process of metadata change

management work in practice

• The benefits of metadata prototype :• Integration with BI environment• Basis for the powerful metadata interface• Standard solution for the metadata repository that allows

flexible customisation of metadata structure, and• Relatively simple support and improvement of the whole

application

15

Conclusions

• Constant business orientation of the metadata solution from the early stages

• Metadata solution should be based on• comprehensive metadata model and • implementation approach, which defines the main steps of

the metadata implementation process.

• Major findings:• Business metadata represents a major part of the

metadata and it is crucial for the Business Intelligence environment

• Majority of the metadata requirements are feasible to implement

16

Conclusions – strategic issues

General issues

Defining the scope for metadata project

Defining the metadata model

Technical issues

Integration with BI environment

Powerful metadata interface

Hierarchic metadata classification

Refreshing metadata from different sources on a regular basis

Metadata quality

Organisational issues

Understanding between business users and technical users

Metadata management and stewardship

17

Acknowledgements

• Supervisors: William Yeoh, Andy Koronios• UniSA Business Intelligence team members: Marc

Conboy, Duncan J Murray, Andrea Matulick and others

• My wife: Olga Ryabova

18

References• Beyer, M. A. (2007). Why Metadata Matters to Business Intelligence Initiatives, Gartner.• Burton, B., J. Popkin, et al. (2007). Workshop Results: Challenges Users Face in Supporting Business

Intelligence, Gartner.• Chisholm, M. (2008). "Business Intelligence Problems and the Abstraction-Translation Paradigm." Retrieved

28/12/2008, 2008.• Gartner. (2007). "Gartner EXP Survey of More than 1,400 CIOs Shows CIOs Must Create Leverage to Remain

Relevant to the Business." Retrieved 01/04/2009, from <http://www.gartner.com/it/page.jsp?id=501189> .• Gartner. (2008). "Gartner EXP Worldwide Survey of 1,500 CIOs Shows 85 Percent of CIOs Expect "Significant

Change" Over Next Three Years." Retrieved 01/04/2009, from <http://www.gartner.com/it/page.jsp?id=587309> .• Gartner. (2009). "Gartner EXP Worldwide Survey of More than 1,500 CIOs Shows IT Spending to Be Flat in

2009." Retrieved 01/04/2009, from <http://www.gartner.com/it/page.jsp?id=855612> .• Foulonneau, M. and J. Riley (2008). Metadata for Digital Resources. Implementation, System Design and

Interoperability. Oxford, Chandos Publishing.• Friedrich, J. R. (2005). Meta-data version and configuration management in multi-vendor environments. ACM

SIGMOD international conference on Management of data, Baltimore, Maryland, ACM.• Inmon, W., B. O'Neil, et al. (2008). Business Metadata, Capturing Enterprise Knowledge, Elsevier.• Lawton, G. (2006). Making Business Intelligence More Useful. Computer, IEEE Computer Society. 39: 14-16.• Paolo Missier, Pinar Alper, et al. (2007) "Requirements and Services for Metadata Management." Semantic

Knowledge Management.• Shankaranarayanan, G. and A. Even (2006). "The metadata enigma." Communications of the ACM 49(2): 88-94.• Vaduva, A. and K. R. Dittrich (2001). Metadata Management for Data Warehousing: Between Vision and Reality.

International Database Engineering & Applications Symposium.• Ward, D. (2007). Data and Metadata Reporting and Presentation Handbook, OECD.