VI SEMESTER

U6CSB25 DATA WAREHOUSING AND MINING

OBJECTIVES

This course has been designed with the following objectives:

To introduce the concept of data mining with in detail coverage of basic tasks, metrics, issues, and implication. Core topics like classification, clustering and association rules are exhaustively dealt with.

To introduce the concept of data warehousing with special emphasis on architecture and design.

UNIT I DATA WAREHOUSING

9

Data warehousing Components Building a Data warehouse Data Mart- DBMS Schemas for Decision Support Data Extraction, Cleanup, and Transformation Tools Metadata.

UNIT II BUSINESS ANALYSIS

9Reporting and Query tools and Applications Tool Categories Cognos Impromptu Online Analytical Processing (OLAP) Need Multidimensional Data Model OLAP Guidelines Multidimensional versus Multirelational OLAP Categories of Tools.UNIT III DATA MINING

9

Introduction Data Types of Data Data Mining Functionalities Interestingness of Patterns Classification of Data Mining Systems Data Mining Task Primitives Integration of a Data Mining System with a Data Warehouse Issues Data Preprocessing.

UNIT IV ASSOCIATION RULE MINING AND CLASSIFICATION

9Mining Frequent Patterns, Associations and Correlations Mining Methods Mining Various Kinds of Association Classification and Prediction - Basic Concepts - Decision Tree

Induction - Bayesian Classification Rule Based Classification Prediction

UNIT V CLUSTERING AND APPLICATIONS AND TRENDS IN DATA MINING 9

Cluster Analysis - Types of Data Categorization of Major Clustering Methods K means Partitioning Methods Hierarchical Methods - Outlier Analysis Data Mining Application Web mining-Text mining-Introduction to big data analytics-Business intelligence.TOTAL: 45+15(Tutorial) = 60 periods TEXT BOOKS

1. Alex Berson and Stephen J. Smith, Data Warehousing, Data Mining & OLAP, Tata

McGraw Hill Edition, Tenth Reprint 2007.

2. Jiawei Han and Micheline Kamber, Data Mining Concepts and Techniques, Second

Edition, Elsevier, 2007.

REFERENCE BOOKS

1. Pang-Ning Tan, Michael Steinbach and Vipin Kumar, Introduction To Data Mining,

Person Education, 2007.

2. K.P. Soman, Shyam Diwakar and V. Ajay , Insight into Data mining Theory and

Practice, Easter Economy Edition, Prentice Hall of India, 2006.

3. G. K. Gupta, Introduction to Data Mining with Case Studies, Easter Economy

Edition, Prentice Hall of India, 2006.

4. Daniel T.Larose, Data Mining Methods and Models, Wile-Interscience, 2006

UNIT I DATA WAREHOUSING

DATA WAREHOUSING COMPONENTSData warehouse:

A data ware house is a subject- oriented, integrated, time variant, and nonvolatile collection of data organized in support of management decision making. Several factors distinguish data warehouses from operational databases. Because the two systems provide quite different functionalities and require different kinds of data, it is necessary to maintain data warehouses separately from operational databases.

The data warehouse is an environment, not a product. It is an architectural construct of an information system that provides users with current and historical decision support information that is hard to access or present in traditional operational data stores.

Overall architecture of Data warehouse:

The data warehouse architecture is based on a relational database management system server that functions as the central repository for informational data. In the data warehouse architecture, operational data and processing is completely separate from data warehouse processing. This central information repository is surrounded by a number of key components designed to make the entire environment functional, manageable, and accessible by both the operational systems that source data into the warehouse and by end user query and analysis tools.

Typically, the source data for the warehouse is coming from the operational applications (an exception might be an operational data store (ODS), discussed briefly in chap.). As the data enters the data warehouse, it is transformed into an integrated structure and formation. The transformation process may involve conversion, summarization, filtering, and condensation of data. Because data within the data warehouse contains a large historical component (sometimes covering 5 to 10 years), the data warehouse must be capable of holding and managing large volumes of data as well as different data structures for the same database over time.

Figure: Data warehouse architecture

The following section takes a close look at the seven major data warehousing components identified in figure.1. Data warehouse Database Relational Database

Transactional Database

Data warehouse

www

2. Data Extraction, Cleanup, Transformation and Migration:

As a component of the data warehouse architecture, proper attention must be given to data extraction, which represents a critical success factor for a data warehouse architecture. Specifically, when implementing data warehouse, several selection criteria that affect the ability to transform, consolidate, integrate, and repair the data should be considered.

The ability to identify data in the data source environments that can be read by the conversion tool is important. This additional step may affect the time liness of data delivery to the warehouse.

Support for flat files, indexed files [e.g., Virtual Storage Access Method (VSAM) and legacy DBMSs [e.g., Information Management System (IMS) and Computer Associates CA IDMS] is critical, since the bulk of corporate data is still maintained in data stores of this type.

The capability to merge data from multiple data stores is required in many installations. Using data replication technologies helps a data change capture facility provides the functionality of reading log data to obtain only the changed data to be transferred. This reduces the amount of data that must be loaded into the data warehouse for periodic updating of the data warehouse.

The specification interface to indicate the data to be extracted and the conversion criteria is important.

The ability to read information from data dictionaries or import information from repository products is desired. This can reduce the amount of specification effort required.

The code generated by the tool should be completely maintainable from within the development environment.

Selective data extraction of both data elements and records enables users to extract only the required data

A field level data examination for the transformation of data into information is needed. for example, a user exit feature is important for users who may need to perform more sophisticated operations on the data than can be achieved using the facilities provided with the tool.

The ability to create summarization, aggregation, and derivation records and fields is very important.

The capability to create summarization, aggregation, and derivation records and fields is very important.

The data warehouse database management system should be able to perform the load directly from the tool, using the native API available with the RDBMS; alternatively, the system should support the capability to create a flat file and then use the load utility provided by the RDBMS.

Vendor stability and support for the product are items that must be carefully evaluated.

3. MetadataMetadata is data about data that describes the data warehouse. It is used for building, maintaining, managing, and using the data warehouse. Metadata can be classified into

Technical metadata, which contains information about warehouse data for use by warehouse designers and administrators when carrying out warehouse development and management tasks. Technical metadata documents include

Information about data sources

Transformation descriptions, i.e., the mapping methods form operational databases into the warehouse, and algorithms used to convert, enhance, or transform data

Warehouse object and data structure definitions for data targets

The rules used to perform data cleanup and data enhancement

Data mapping operations when capturing data from source systems and applying it to the target warehouse database.

Access authorization, backup history, archive history, information delivery history, data acquisition history, data access, etc.

Business metadata contains information that gives users an easy to understand perspective of the information stored in the data warehouse. Business metadata and information object type, including queries, reports, images, video, and / or audio clips

Subject areas and information object type, including queries, reports, images, video, and or audio clips.

Internet home pages

Other information to support all data warehousing components. For example, the information related to the information delivery system should include subscription information, scheduling information, details of delivery destinations, and the business query objects such as predefined queries, reports, and analyses.

Data warehouse operational information, e.g., data history (snapshots, versions,) ownership, extract audit trail, usage data

Equally important, metadata provides interactive access to users to help understand content, find data. One of the issues dealing with metadata relates to the fact that many data extraction tool capabilities to gather metadata remain fairly immature. Therefore, often there is a need to create a metadata interface for users, which may involve some duplication of effort.

Metadata management is provided via a metadata repository and accompanying software. Metadata repository management software can be used to map the source data to the target database, generate code for data transformations, integrate and transform the data, and control moving data the warehouse. This software which typically runs on a workstation, enables users to specify how the data should be transformed, such as data mapping, conversion, and summarization.

As users interactions with the data warehouse increase, their approaches to reviewing the results of their requests for information can be expected to evolved from relatively manual analysis for trends and exceptions to agent driven initiation of the analysis based on user defined thresholds. The definition of these thresholds, configuration parameters for the software agents using them, and the information directory indicating where the appropriate sources for the required information can be found are all stored in the metadata repository as well.

One of the important functional components of the metadata repository is the information directory. The content of the information directory is the metadata that helps technical and business users exploit the power of data ware housing. This directory helps integrate, maintain, and view the contents of the data warehousing system.

From a technical requirements point of view, the information directory and the entire metadata repository.

Should be a gateway to the data warehouse environment, and thus should be accessible from any platform via transparent and seamless connections

Should support an easy distribution and replication of its content for high performance and availability

Should be searchable by business oriented key words

Should act as a launch platform for end user data access and analysis tools

Should support the sharing of information objects such as queries, reports, data collections, and subscriptions between users.

Should support a variety of scheduling options for requests against the data warehouse, including on demand, one time, repetitive, event driven, and conditional delivery (in conjunction with the information delivery system)

Should support the distribution of the query results to one or more destinations in any of the user specified formats (in conjunction with the information delivery system)

Should support and provide interfaces to other applications such as e mail, spreadsheet, and schedulers

4. Access Tools:

Currently, no single tool on the market can handle all possible data warehouse access needs. Therefore, most implementations rely on a suite of tools. The best way to choose this suite includes the definition of different types of access to the data and selecting the best tool for that kind of access. Examples of access types include.

Simple tabular form reporting

Ranking

Multivariable analysis

Time series analysis

Data visualization, graphing, charting, and pivoting

Complex textual search

Statistical analysis

Artificial intelligence techniques for testing of hypothesis, trends discovery, definition, and validation of data clusters and segments.

Information mapping (i.e., mapping of spatial data in geographic information systems)

Ad hoc user specified queries

Predefined repeatable queries

Interactive drill down reporting and analysis

Complex queries with multitable joins, multilevel subqueries, and sophisticated search criteria.

In addition, certain business requirements often exceed existing tool capabilities and may require building sophisticated applications to retrieve and analyze warehouse data. These applications often take the form of custom developed screens and reports that retrieve frequently used data and format it in a predefined standardized way. This approach may be very useful for those data warehouse users who are not yet comfortable with ad hoc queries.

There are a number of query tools on the market today. Many of these tools are designed to easily compose and execute ad hoc queries and build customized reports with little knowledge of the underlying database technology, SQL, or even the data model (i.e., Impromptu from Cognos, Business Objects, etc.), while others (e.g., Andynes GQL) provide relatively low level capabilities for an expert user to develop complex ad hoc queries in a fashion similar to developing SQL queries for relational databases. Business requirements that exceed the capabilities of ad hoc query and reporting tools are fulfilled by different classes of tools: OLAP and data mining tools. The technology behind these tools is discussed later in the book.

5. Query and Reporting Tools:

This category can be further divided into two groups: reporting tools and managed query tools.

Reporting tools can be divided into production reporting tools and desktop report writers.

Production reporting tools will let companies generate regular operational reports or support high volume batch jobs, such as calculating and printing paychecks.

Report writers, on the other hand, are inexpensive desktop tools designed for end users.

Managed query tools shield end users from the complexities of SQL and data base structures by inserting a metalayer between users and the database. The metalayer is the software that provides subject oriented views of a database and supports point and click creation of SQL. These tools are designed for easy to use, point and click, and visual navigation operations that either accept SQL or generate SQL statements to query relational data stored in the warehouse. Some of these tools proceed to format the retrieved data into easy to read reports, while others concentrate on the on screen presentation. These tools are the preferred choice of the users of business applications such as segment identification, demographic analysis, territory management, and customer mailing lists. As the complexity of the questions grows, these tools may rapidly become inefficient.

6. Data Marts:

The concept of the data mart is causing a lot of excitement and attracts much attention in the data warehouse industry. Mostly, data marts are presented as an inexpensive alternative to a data warehouse that takes significantly less time and money to build. However, The term data mart means different things to different people. A rigorous definition of this term is a data store that is subsidiary to a data warehouse of integrated data. The data mart is directed at a partition of data (often called a subject area) that is created for the use of a dedicated group of users. A data mart might, in fact, be a set of denormalized, summarized, or aggregated data. Sometimes, such a set could be placed on the data warehouse database rather than a physically separate store of data. In most instances, however, the data mart is a physically separate store of data and is normally resident on a separate database server, often on the local area network serving a dedicated user group. Sometimes the data mart simply comprises relational OLAP technology which creates highly denormalized start schema relational designs or hyper cubes of data for analysis by groups of users with a common interest in a limited portion of the database. In other cases, the data warehouse architecture may incorporate data mining tools that extract sets of data for a particular type of analysis. All these types of data marts, called dependent data marts because their data content is sourced from the data warehouse, have a higher value because no matter how many are developed and no matter how many are deployed and no matter how many different enabling technologies are used, the different users are all accessing the information views derived from the same single integrated version of the data.

Unfortunately, the misleading statements about the simplicity and low cost of data marts sometimes result in organizations or vendors incorrectly positioning them as an alternative to the data warehouse. This viewpoint defines independent data marts that in fact represent fragmented point solutions to a range of business problems in the enterprise. This type of implementation should rarely be deployed in the context of an overall technology or applications architecture. Indeed, it is missing the ingredient that is at the heart of the data warehousing concept: data integration. Each independent data mart makes its own assumptions about how to consolidate the data, and the data across several data marts may not be consistent.

Moreover, the concept of an independent data mart is dangerous as soon as the first data mart is created, other organizations, groups, and subject areas within the enterprise embarks on the task of building their own data marts. As a result, you create an environment in which multiple operational systems feed multiple nonintegrated data marts that are often overlapping in data content, job scheduling, connectivity, and management. In other words, you have transformed a complex many to one problem of building a data warehouse from operational and external data sources to a many to many sourcing and management nightmare.

Another consideration against independent data marts is related to the potential scalability problem. The first simple and inexpensive data mart was most probably designed without any serious consideration about the scalability (for example, you wouldnt consider an expensive parallel computing platform for an inexpensive and small data mart). But, as usage begets usage, the initial small data mart needs to grow (i.e., in data sizes and the number of concurrent users), without any ability to do so in a scalable fashion.

We would like to make it clear that the point solution independent data mart is not necessarily a bad thing, and it is often a necessary and valid solution to a pressing business problem, thus achieving the goal of rapid delivery of enhanced decision support functionality to end users. The business drivers underlying such development include.

Extremely urgent user requirements

The absence of a budget for a full data warehouse strategy

The absence of a sponsor for an enterprise wide decision support strategy

The decentralization of business units

The attraction of easy to use tools and a mind sized project

7.Data Mining:

A critical success factor for any business today is its ability to use information effectively. This strategic use of data can result from opportunities presented by discovering hidden, previously undetected, and frequently extremely valuable facts about consumers, retailers and suppliers, business trends, and direction and significant factors. Knowing this information, an organization can formulate effective business, marketing, and sales strategies; precisely target promotional activity; discover and penetrate new markets; and successfully compete in the marketplace form a position of informed strength. A relatively new and promising technology aimed at achieving this strategic advantage is known as data mining.

Lets define Data mining as the process of discovering meaningful new correlations, patterns, and trends by digging into (mining) large amounts of data stored in warehouses, using artificial intelligence (AI) and statistical and mathematical techniques. In these areas, data mining can reach beyond the capabilities of OLAP, especially since the major attraction of data mining is its ability to build predictive rather than retrospective models.

Data mining is not specific to any industry it requires intelligent technologies and the willingness to explore the possibility of hidden knowledge that resides in the data. Industries that are already taking advantage of data mining include retail, financial, medical, manufacturing, environmental, utilities, security, transportation, chemical, insurance, and aerospace industries, with the early success stories coming primarily from the retail, financial, and medical sectors. The organizations using data mining techniques report gaining insights into their respective businesses by revealing implicit relationships, patterns, surprisingly significant facts, trends, exceptions, and anomalies previously unavailable through the human analysts. These experiences show that, although data mining is still an emerging discipline, it has a huge potential to gain significant benefits in the marketplace. Most organizations engage in data mining to

Discover knowledge. The goal of knowledge discovery is to determine explicit hidden relationship, patterns, or correlations from data stored in an enter prises database. Specifically, data mining can be used to perform:

Segmentation (e.g., group customer records for custom tailored marketing)

Classification (assignment of input data to a predefined class, discovery) and understanding of trends, text document classification)

Association (discovery of cross sales opportunities)

Prediction (determining preference of customers majority)

Visualize data. Analysts must make sense out of a huge amount of information stored in corporate databases. Prior to any analysis, the goal is to humanize the mass of data they must deal with and find a clever way to display the data.

Correct data. While consolidating massive databases, many enterprises find that the data is not complete and invariably contains erroneous and contradictory information. Data mining techniques can help identify and correct problems in the most consistent way possible. The number of applications within this category is somewhat limited because of the difficult nature of the correction process. Replacing missing values or correcting what can be perceived as wild values requires judgment calls that are difficult to provide automatically.

8. OLAPOn line analytical processing (OLAP) tools. These tools are based on the concepts of multidimensional databases and allow a sophisticated user to analyze the data using elaborate, multidimensional, complex views.

9. Information Delivery System

The information delivery system distributes warehouse stored data and other information objects to other data warehouses and end user products such as spreadsheets and local databases. Delivery of information may be based on time of day, or on a completion of an external event.

BENEFITS OF DATA WAREHOUSING:Data warehouse usage includes

Locating the right information

Presentation of information

Testing of hypothesis

Discovery of information

Sharing the analysis

Using better tools to access data can reduce outdated, historical data. Like wise, users can obtain the data when they need it most, often during business decision processes, not on a schedule predetermined months earlier by the IS department and computer operations staff.

Data warehouse architecture can enhance overall availability of business intelligence data, as well as increase the effectiveness and timeliness of business decisions.

Tangible benefits:

Successfully implemented data warehousing can realize some significant tangible benefits. For example, conservatively assuming an improvement in out of stock conditions in the retailing business that leads to a 1 percent increase in sales can mean a sizable cost benefit (e.g., even for a small retail business with $200 million in annual sales, a conservative 1 percent improvement in sales can yield additional annual revenue of $2 million or more). In fact, several retail enterprises claim that data warehouse implementations have improved out of stock conditions to the extent that sales increases range from 5 to 20 percent. This benefits is in addition to retaining customers who might not have returned if, because of out of stock problems, they had to do business with other retailers.

Other examples of tangible benefits of a data warehouse initiative include the following:

Product inventory turnover is improved

Costs of product introduction are decreased with improved selection of target markets

More cost effective decision making is enabled by separating (ad hoc) query processing from running against operational databases.

Better business intelligence is enabled by increased quality and flexibility of market analysis available through multilevel data structures, which may range from detailed to highly summarize. For example, determining the effectiveness of marketing programs allows the elimination of weaker programs and enhancement of stronger ones.

Enhanced asset and liability management means that a data warehouse can provide a big picture of enterprise wide purchasing and inventory patterns, and can indicate otherwise unseen credit exposure and opportunities for cost savings.

Intangible benefits:

In addition to the tangible benefits outlined above, a data warehouse provides a number of intangible benefits. Although they are more difficult to quantify, intangible benefits should also be considered when planning for the data warehouse. Examples of intangible benefits are:

Improved productivity, by keeping all required data in a single location and eliminating the rekeying of data.

Reduced redundant processing, support, and software to support over lapping decision support applications.

Enhanced customer relations through improved knowledge of individual requirements and trend, through customization, improved communications, and tailored product offerings.

Enabling business process reengineering data warehousing can provide useful insights into the work processes themselves, resulting in developing breakthrough ideas for the reengineering of those processes.

BUILDING A DATA WAREHOUSE

Design Consideration to Build A Data Warehouse:

To be successful, a data warehouse designer must adopt a holistic approach consider all data warehouse components as parts of a single complex system and take into the account all possible data sources and all known usage requirements. Failing to do so may easily result in a data warehouse design that is skewed toward a particular business requirement, a particular data source, or a selected access tool.

In general a data warehouses design point is to consolidate data from multiple, often heterogeneous, sources into a query database. This is also one of the reasons why a data warehouse is rather difficult to build.

The main factors include

Heterogeneity of data sources, which affects data conversion, quality, time lines

Use of historical data, which implies that data may be old

Tendency of databases to grow very large

Another important point concerns the experience and accepted practices. Basically, the reality is that the data warehouse design is different from traditional OLTP. Indeed, the data warehouse is business driven (not IS driven, as in OLTP), requires continuous interactions with end users, and is never finished, since both requirements and data sources change. Understanding these points allows developers to avoid a number of pitfalls relevant to data warehouse development, and justifies a new approach to data warehouse design: a business driven, continuous iterative warehouse engineering approach. In addition to these general considerations, there are several specific points relevant to the data warehouse design.

Data content:

One common misconception about data warehouses is that they should not contain as much detail level data as operational systems used to source this data in. In reality, however, while the data in the warehouse is formatted differently from the operational data, it may be just as detailed. Typically, a data warehouse may contain detailed data, but the data is cleaned up and transactional attributes of the data are filtered out. These attributes are mostly the ones used for the internal transaction system logic, and they are not meaningful in the context of analysis and decision making.

The content and structure of the data warehouse are reflected in its data model. The data model is the template that describes how information will be organized within the integrated warehouse framework. It identifies major subjects and relationships of the model, including keys, attribute, and attribute groupings. In attrition, a designer should always remember that decision support queries, because of their broad scope and analytical intensity, require data models to be optimized to improve query performance. In addition to its effect on query performance, the data model affects data storage requirements and data loading performance.

Additionally, the data model for the data warehouse may be (and quite often is) different from the data models for data warehouse, and may contain highly aggregated and summarized data in the form of a specialized denor-malized relational schema (star schema) or as a multidimensional data cube. The key point is, however, that in a dependent data mart environment, the data mart data is cleaned up, is transformed, and is consistent with the data warehouse and other data marts sourced from the same warehouse.

Metadata:

As already discussed, metadata defines the contents and location of data (data model) in the warehouse, relationships between the operational databases and the data warehouse, and the business views of the warehouse data that are accessible by end user tools. Metadata is searched by users to find data definitions or subject areas. In other words, metadata provides decision support oriented pointers to warehouse data, and thus provides a logical link between warehouse data and the decision support application. A data warehouse design should ensure that there is a mechanism that populates and maintains the metadata repository, and that all access paths to the data warehouse have metadata as an entry point. To put it another way, the warehouse design should prevent any direct access to the warehouse data (especially updates) if it does not use metadata definitions to gain the access.

Data distribution:

One of the biggest challenges when designing a data warehouse is the data placement and distribution strategy. This follows from the fact that as the data volumes continue to grow, the database size may rapidly outgrow a single server. Therefore, it becomes necessary to know how the data should be divided across multiple servers, and which users should get access to which types of data. The data placement and distribution design should consider several options, including data distribution by subject area (e.g., human resources, marketing), location (e.g., geographic regions), or time (e.g., current, monthly, quarterly). The designers should be aware that, while the distribution solves a number of problems, it may also create a few of its own; for example, if the warehouse servers are distributed across multiple locations, a query that spans several servers across the LAN or WAN may flood the network with a large amount of data. Therefore, any distribution strategy should take into account all possible access needs for the warehouse data.

Tools:

A number of tools available today are specifically designed to help in the implementation of a data warehouses. These tools provide facilities for defining the transformation and cleanup rules, data movement (from operational sources into the warehouses), end user query, reporting, and data analysis. Each tool takes a slightly different approach to data warehousing and often maintains its own version of the metadata which is placed in a tool specific, proprietary metadata repository. Data warehouse designers have to be careful not to sacrifice the overall design to fit a specific tool. At the same time, the designers have to make sure that all selected tools are compatible with the given data warehouse environment and with each other. That means that all selected tools can use a common metadata repository. Alternatively, the tools should be able to sources the metadata from the warehouse data dictionary (if it exists) or from a CASE tool used to design the warehouse database. Another option is to use metadata gateways that translate one tools metadata into another tools format. If these requirements are not satisfied, the resulting warehouse environment may rapidly become unmanageable, since every modification to the warehouse data model may involve some significant and labor intensive changes to the metadata definitions for every tool in the environment. And then, these changes would have to be verified for consistency and integrity.

Performance considerations:

Although the data warehouse design point does not include sub second response times typical of OLTP systems, it is nevertheless a clear business requirement that an ideal data warehouse environment should support inter active query processing. In fact, the majority of end user tools is designed as interactive applications. Therefore, rapid query processing is a highly desired feature that should be designed into the data warehouse. Of course, the actual performance levels are business dependent and vary widely from one environment to another. Unfortunately, it is relatively difficult to predict the performance of a typical data warehouse. One of the reasons for this is the unpredictable usage patterns against the data. Thus, traditional database design and tuning techniques dont always work in the data warehouse arena. When designing a data warehouse. Therefore, the need to clearly understand users informational requirements becomes mandatory. Specifically, knowing how end users need to access various data can help design warehouse data bases to avoid the majority of the most expensive operations such as multitable scans and joins. For example, one design technique is to populate the warehouse with a number of denormalized views containing summarized, derived, and aggregated data. If done correctly, many end user queries any execute directly against these views, thus maintaining appropriate overall performance levels.

Nine decisions in the design of a data warehouse:

The job of a data warehouse designer is a daunting one. Often the newly appointed data warehouse designer is drawn to the job because of the high visibility and importance of the data warehouse function. In effect, management says to the designer. Take all the enterprise data and make it available to management so that they can answer all their questions and sleep at night. A please do it very quickly, and were sorry, but we cant add any more staff until the proof of concept is successful.

This responsibility is exciting and very visible, but more designers feel over whelmed by the sheer enormity of the task. Something real needs to be accomplished, and fast. Where do you start? Which data should be brought up first? Which management needs are most important? Does the design depend on the details of the most recent interview, or are there some underlying and more constant design guidelines that you can depend on? how do you scope the project down to something manageable, yet at the same time build an extensible architecture that will gradually let you behind a comprehensive data warehouse environment?

These questions are close to causing a crisis in the data warehouse industry. Much of the recent surge in the industry toward data marts is a reaction to these very issues. Designers want to do something simple and achievable. No one is willing to sign up for a galactic design that must somehow get everything right on the first try. Everyone hopes that in the rush to simplification the long-term coherence and extendibility of the design will not be compromised. Fortunately, a pathway through this design challenge achieves an implementable immediate result, and at the same time it continuously augments the design so that eventually a true enterprise-scale data warehouse is built. The secret is to keep in mind a design methodology, which Ralph Kimball calls the nine-step method

As a result of interviewing marketing users, finance users, salesforce users, operational users, first and second level management, and senior management, a picture emerges of what is keeping these people awake at night. You can list and prioritize the primary business issues facing the enterprise. At the same time, you should conduct a set of interviews with the legacy systems DBAs, who will reveal which data sources are clean, which contain valid and consistent data, and which will remain supported over the next few years.

Preparing for the design with a proper set of interviews is crucial. Interviewing is also one of the hardest things to teach people. I find it helpful to reduce the interviewing process to a tactic and an objective. Crudely put, the tactic is to make the end users talk about what they do, and the objective is to gain insights that feed the nine design decisions. The tricky part is that the interviewer cant pose the design questions directly to the end users. End users dont have opinions about data warehouse system design issues; they have opinions about what is important in their business lives. End users are intimidated by system design questions, and they are quite right when they insist that system design is IT responsibility, not theirs. Thus, the challenge of the data mark designer is to meet the users far more than half way.

In any event, armed with both the top-down view (what keeps management awake) and the bottom-up view (which data sources are available), the data warehouse designer may follow these steps.

TABLE 7.1: Nine-Step Method in the Design of a Data Warehouse

1. Choosing the subject matter.

2. Deciding what a fact table represents.

3. Identifying and conforming the dimensions.

4. Choosing the facts

5. Storing pre-calculations in the fact table.

6. Rounding out the dimension tables.

7. Choosing the duration of the data base.

8. The need to track slowly changing dimensions.

9. Deciding the query priorities and the query modes.

Steps 1: Choosing the subject matter of a particular data mart. The first data mart you build be the one with the most bang for the buck. It should simultaneously answer the most important business questions and be the most accessible in terms of data extraction. According to Kimball, a great place to start in most enterprises is to build a data mart that consists of customer invoices or monthly statements. This data source is probably fairly accessible and of fairly high quality. One of Kimballs laws is that the best data source in any enterprise is the record of how much money they us. Unless costs and profitably are easily available before the data mart is even designed, its best to avoid adding these items to this first data mart. Nothing drags down a data mart implementation faster than a heroic or impossible mission to provide activity-based costing as part of the first deliverable.

Step 2: deciding exactly what a fact table record represents. This step, according to R. Kimball, seems like a technical detail at this early point, but it is actually the secret to making progress on the design. As explained in chapter, the fact table is the large central table in the dimensional design that has a multipart key. Each component of the multipart key is a foreign key to an individual dimension table. In the example of customer invoices, the grain of the fact table is the individual line item on the customer invoice. In other words, a line item on an invoice is a single fact table record, and vice versa. Once the fact table representation is decided, a coherent discussion of what the dimensions of the data marts fact table are can take place.

Step 3: identifying and conforming the dimensions. The dimensions are the drivers of the data mart. The dimensions are the platforms for browsing the allowable constraint values and launching these constraints. The dimensions are the source of row headers in the users final report; they carry the enterprises vocabulary to the users. A well-architected set of dimensions the enterprises vocabulary to the users. A well-architected set of dimensions makes the data mart understandable and easy to use. A poorly presented or incomplete set of dimensions robs the data mart of its usefulness. Dimensions should be chosen with the long-range data warehouse in mind. This choice presents the primary moment at which the data mart architect must disregard the data mart details and consider the longer-range plans. If any dimension occurs in two data marts, they must be exactly the same dimension, or one occurs in two data marts, they must be exactly the same dimension, or one must be a mathematical subset of the other. Only in this way two data marts share one or more dimensions in the same application. When a dimension is used in two data marts, this dimension is said to be confirmed. Good examples of dimensions that absolutely must be conformed between data marts are the customer and product dimensions in an enterprise. If these dimensions are allowed to drift out of synchronization between data marts, the overall data warehouse will fail, because the two data marts will not be able to be used together. The requirement to conform dimensions across data marts is very strong. Careful though must be given to this requirement before the first data mart is implemented. The data mart team must figure out what an enterprise product ID is. If this task is done correctly, successive data marts can be built at different times, on different machines, and by different development teams, and these data marts will merge coherently into an overall data warehouse. In particular, if the dimensions of two data marts are conformed, it is easy to implement drill across by sending separate queries to the two data marts, and then sort-merging the two answer sets on a set of common row headers. The row headers can be made to be common only if they are drawn from a conformed dimension common to the two data marts.

Technical Consideration to Build A Data Warehouse:

A number of technical issues are to be considered when designing and implementing a data warehouse environment. These issues include

The hardware platform that would house the data warehouse

The database management system that supports the warehouse database

The communications infrastructure that connects the warehouse, data marts, operational systems, and end users.

The hardware platform and software to support the metadata repository

The systems management framework that enables centralized management and administration of the entire environment.

Lets look at some of these issues in more detail.

Hardware platforms:

Since many data warehouse implementations are developed into already existing environments, many organizations tend to leverage the existing platforms and skill base to build a data warehouse. This section looks at the hardware platform selection from an architectural viewpoint: what platform is best to build a successful data warehouse from the ground up.

An important consideration when choosing a data warehouse server is its capacity for handling the volumes of data required by decision support applications, some of which may require a significant amount of historical (e.g., up to 10 years) data. This capacity requirement can be quite large. For example, in general, disk storage allocated for the warehouse should be 2 to 3 times the size of the data component of the warehouse to accommodate DSS processing, such as sorting, storing of intermediate results, summarization, join, and formatting. Often, the platform choice is the choice between a mainframe and non MVS (UNIX or Windows NT) server.

Of course, a number of arguments can be made for and against each of these choices. For example, a mainframe is based on a proven technology: has large data and throughput capacity; is reliable, available, and serviceable; and may support the legacy databases that are used as sources for the data warehouse. The data warehouse residing on the mainframe is best suited for situations in which large amounts of legacy data need to be stored in the data warehouse. A mainframe system, however, is not as open and flexible as a contemporary client / server (non mainframe) can also support large data volumes and a large number of flexible GUI based end user tools, and can relieve the mainframe from ad hoc query processing. However, in general, non MVS servers are not as reliable as mainframes, are more difficult to manage and integrate into the existing environment, and may require new skills and even new organizational structures.

From the architectural viewpoint, however, the data warehouse server has to be specialized for the tasks associated with the data warehouse, and a mainframe can be well suited to be a data warehouse server. Lets look at the hard ware features that make a server whether it is mainframe- , UNIX-, or NT based an appropriate technical solution for the data warehouse.

The begin with; the data warehouse server has to be able to support large data volumes and complex query processing. In addition, it has to be scalable, since the data warehouse is never finished, as new user requirements, new data sources, and more historical data are continuously incorporated into the warehouse, and as the user population of the data warehouse continues to grow. Therefore, a clear requirement for the data warehouse server is the scalable high performance for data loading and ad hoc query processing as well as the ability to support large databases in a reliable, efficient fashion. Chapter briefly touched on various design points to enable server specialization for scalability in performance, throughput, user support, and very large database (VLDB) processing.

Balanced approach:

An important design point when selecting a scalable computing platform is the right balance between all computing components, for example, between the number of processors in a multiprocessor system and the I/O bandwidth. Remember that the lack of balance in a system inevitably results in a bottleneck!

Typically, when a hardware platform is sized to accommodate the data ware house, this sizing is frequently focused on the number and size of disks. A typical disk configuration includes 2.5 to 3 times the amount of raw data. An important consideration disk throughput comes from the actual number of disks, and not the total disk space. Thus, the number of disks has direct impact on data parallelism. To balance the system, it is very important to allocate a correct number of processors to efficiently handle all disk I/O operations. If this allocation is not balanced, an expensive data warehouse platform can rapidly become CPU- bound. Indeed, since various processors have widely of disks per CPU, data warehouse designers should carefully analyze the disk I/O rates and processor capabilities to derive an efficient system configuration. For example if it takes a CPU rated at 10 SPECint processor in a multiprocessor system can handle three disk drives. Knowing how much data needs to be processed should give you an idea of how big the multiprocessor system should be. Another consideration is related to disk controllers. A disk controller can support a certain amount of data throughput (e.g., 20 Mbytes/s). Knowing the per disk throughput ratio and the total number of disks can tell you how many controllers of a given type should be configured in the system.

The idea of a balanced approach can (and should) be carefully extended to all system components. The resulting system configuration will easily handle known workloads and provide a balanced and scalable computing platform for future growth.

Optimal hardware architecture for parallel query scalability:

An important consideration when selecting a hardware platform for a data warehouse is that of scalability. Therefore, a frequent approach to system selection is to take advantage of hardware parallelism that comes in the form of shared memory symmetric multiprocessors (SMPs), clusters, and shared nothing distributed memory systems (MPPs). As was shown in chap. 3, the scalability of these systems can be seriously affected by the system architectures induced data skew. This architecture induced data skew is more severe in the low density asymmetric connection architectures (e.g., daisy chained, 2- D and 3 D mesh), and is virtually nonexistent in symmetric connection architectures (e.g., cross bar switch). Thus, when selecting hardware platform for a data skew can overpower even the best data layout for parallel query execution, and can force an expensive parallel computing system to process queries serially.

Data warehouse and DBMS specialization:

To reiterate, the two important challenges facing the developers of data warehouses are the very large size of the databases and the need to process complex ad hoc queries in a relatively short time. Therefore, among the most important requirements for the data warehouse DBMS are performance, throughput, and scalability.

The majority of established RDBMS vendors has implemented various degrees of parallelism in their respective products. Although any relational database management system such as DB2, Oracle, Informix, or Sybase supports parallel database processing, some of these products have been architected to better suit the specialized requirements of the data warehouse.

In addition to the traditional relational DBMSs, there are databases that have been optimized specifically for data warehousing, such as Red Brick Warehouse form Red Brick Systems. The BDMS features designed to satisfy the high performance and scalability requirements of a data warehouse are briefly discussed in chapter. The next chapter looks into the DBMS characteristics for performance and scalability from the data warehouse viewpoint.

Communications infrastructure:

When planning for a data warehouse, one often neglected aspect of the architecture is the cost and efforts associated with bringing access to corporate data directly to the desktop. These costs and efforts could be significant, since many large organizations do not have a large user population with direct electronic access to information, and since a typical data warehouse user requires a relatively large bandwidth to interact with the data warehouse and retrieve a significant amount of data for the analysis. This may mean that communications networks have to be expanded, and new hardware and software may have to be purchased.

Implementation Considerations to build a data warehouse:

A data warehouse cannot be simply bought and installed its implementation requires the integration of many products within a data warehouse. The caveat here is that the necessary customization drives up the cost of implementing a data warehouse. To illustrate the complexity of the data warehouse implementation. Lets discuss the logical steps needed to build a data warehouses:

Collect and analyze business requirements.

Create a data model and a physical design for the data warehouse.

Define data sources.

Choose the database technology and platform for the warehouse.

Extract the data from the operational databases, transform it, clean it up, and load it into the database.

Choose database access and reporting tools.

Choose database connectivity software

Choose data analysis and presentation software

Update the data warehouse.

When building the warehouse, these steps must be performed within the constraints of the current state of data warehouse technologies.

Access Tools:

Currently, no single tool on the market can handle all possible data warehouse access needs. Therefore, most implementations rely on a suite of tools. The best way to choose this suite includes the definition of different types of access to the data and selecting the best tool for that kind of access. Examples of access types include.

Simple tabular form reporting

Ranking

Multivariable analysis

Time series analysis

Data visualization, graphing, charting, and pivoting

Complex textual search

Statistical analysis

Artificial intelligence techniques for testing of hypothesis, trends discovery, definition, and validation of data clusters and segments.

Information mapping (i.e., mapping of spatial data in geographic information systems)

Ad hoc user specified queries

Predefined repeatable queries

Interactive drill down reporting and analysis

Complex queries with multitable joins, multilevel sub queries, and sophisticated search criteria.

In addition, certain business requirements often exceed existing tool capabilities and may require building sophisticated applications to retrieve and analyze warehouse data. These applications often take the form of custom developed screens and reports that retrieve frequently used data and format it in a predefined standardized way. This approach may be very useful for those data warehouse users who are not yet comfortable with ad hoc queries.

There are a number of query tools on the market today. Many of these tools are designed to easily compose and execute ad hoc queries and build customized reports with little knowledge of the underlying database technology, SQL, or even the data model (i.e., Impromptu from Cognos, Business Objects, etc.), while others (e.g., Andynes GQL) provide relatively low level capabilities for an expert user to develop complex ad hoc queries in a fashion similar to developing SQL queries for relational databases. Business requirements that exceed the capabilities of ad hoc query and reporting tools are fulfilled by different classes of tools: OLAP and data mining tools. The technology behind these tools is discussed later in the book.

Data Extraction, Cleanup, Transformation and Migration:

As a component of the data warehouse architecture, proper attention must be given to data extraction, which represents a critical success factor for a data warehouse architecture. Specifically, when implementing data warehouse, several selection criteria that affect the ability to transform, consolidate, integrate, and repair the data should be considered.

The ability to identify data in the data source environments that can be read by the conversion tool is important. This additional step may affect the time liness of data delivery to the warehouse.

Support for flat files, indexed files [e.g., Virtual Storage Access Method (VSAM) and legacy DBMSs [e.g., Information Management System (IMS) and Computer Associates CA IDMS] is critical, since the bulk of corporate data is still maintained in data stores of this type.

The capability to merge data from multiple data stores is required in many installations. Using data replication technologies helps a data change capture facility provides the functionality of reading log data to obtain only the changed data to be transferred. This reduces the amount of data that must be loaded into the data warehouse for periodic updating of the data warehouse.

The specification interface to indicate the data to be extracted and the conversion criteria is important.

The ability to read information from data dictionaries or import information from repository products is desired. This can reduce the amount of specification effort required.

The code generated by the tool should be completely maintainable from within the development environment.

Selective data extraction of both data elements and records enables users to extract only the required data

A field level data examination for the transformation of data into information is needed. for example, a user exit feature is important for users who may need to perform more sophisticated operations on the data than can be achieved using the facilities provided with the tool.

The ability to create summarization, aggregation, and derivation records and fields is very important.

The capability to create summarization, aggregation, and derivation records and fields is very important.

The data warehouse database management system should be able to perform the load directly form the tool, using the native API available with the RDBMS; alternatively, the system should support the capability to create a flat file and then use the load utility provided by the RDBMS.

Vendor stability and support for the product are items that must be carefully evaluated.

Vendor Solutions:

Some vendors have emerged that are more focused on fulfilling requirements pertaining to data warehouse implementations as opposed to simply moving data between hardware platforms.

The extraction tools briefly described in the following paragraphs illustrate three separate approaches to the warehousing extraction function. Prism markets a primarily model based approach, while Information Builders markets a gateway approach. SAS products could handle all the warehouse functions, including extraction. Several extraction tools are discussed in more detail in chapter.

Prism solutions:

Prism Warehouse Manager maps source data to a target database management system to be used as a warehouse. Warehouse Manager generates code to extract and integrate data, create and manage metadata, and build a subject oriented, historical base. The standard coversions, key changes, structural changes, and condensations needed to transform operational data into data warehouse information are automatically created. Prism Warehouse Manager can extract data from multiple source environments, including DBS, IDMS, IMS, VSAM, RMS and sequential files under UNIX or MVS. Target databases include Oracle, Sybase, and Informix. Prism Solutions has strategic relationships with Pyramid and Informix.

Carletons PASSPORT:

PASSPORT is positioned in the data extract and transformation niche of data warehousing. The product currently consists of two components. The first, which is mainframe based, collects the file record table layouts for the required inputs and outputs and converts them to the Passport Data Language (PDL). The second component is workstation based and is used to create the metadata directory from which it builds the COBOL programs to actually create the extracts. The user must transfer the PDL file from the main frame to a location accessible by PASSPORT. The metadata directory is stored in a relational database (e.g., DB2 Common Server). The product itself runs on Windows and OS /2. The Windows shows a graphical representation of the workflow that operates using the standard expand, collapse, drill down, etc., metaphor. Carletons PASSPORT can produce multiple output files from a single execution of an extract program. Carleton is partnering with Sybase on the integration of PASSPORT with SYBASE Open Client and Open Server.

Information Builders Inc:

The EDA / SQL family of products provides SQL access to and a uniform relational view of relational and non relational data residing in over 60 different databases on 35 different platforms. EDA / SQL implements a client / server model that is optimized for higher performance. EDA / SQL supports copy management, data quality management, data replication capabilities, and standards support for both ODBC and the X/Open CLI. IBIs product is the component of almost every systems suppliers legacy data access strategy, including Amdahl, IBM, Digital Equipment, HP, and Bull as well as many database and tool vendors.

SAS Institute Inc:

SAS begins with the premise that most mission critical data still resides in the data center and offers its traditional SAS Systems tools as a means to serve all data warehousing functions. Its data repository function can act to build the informational database. SAS Data Access Engines serve as extraction tools to combine common variables, transform data representation forms for consistency, consolidate redundant data, and use business rules to produce computed values in the warehouse. SAS views server the internetworking and refresh roles, and SAS reporting, graphing, and decision support products act as the front end. In addition to interacting with SAS System databases, SAS engines can work with hierarchical and relational databases and sequential files.

Data Placement Strategies:

As a data warehouse grows, there are at least two options for data placement. One is to put some of the data in the data warehouse into another storage media; e.g., WORM, RAID, or photo optical technology. The data selected for transport to the alternate storage media is detailed and older and there is less demand for it. The bulk storage can be handled either by the data warehouse server or another server used exclusively for handling the bulk storage media.

The second option is to distribute the data in the data warehouse across multiple servers. Some criteria must be established for dividing it over the servers by geography, organization unit, time, function, etc. Another factor in determining how data should be divided among multiple data warehouse servers is the pattern of usage, such as what data is accessed and how much data is accessed, and any joins, sorts, or other processing that occurs after the data is accessed. However the data is divided, a single source of metadata across the entire organization is required. Hence, this configuration requires both corporation wide metadata and the metadata managed for any given server.

Data replication:

Where most users require only a small subset of the corporate data, using data replication or data movers to place only the data that is relevant to a particular workgroup in a localized database can be a more affordable solution than data warehousing. In many cases, often only a small portion of the data a company collects will be relevant. Many companies use data replication servers to copy their most needed data to a separate data base where decision support applications can access it. Replication technology creates copies of data bases on a periodic basis, so that data entry and data analysis can be performed separately. Thus, end users ad hoc queries and analytical processing are prevented from competing with operational applications for server resources.

Similar process can be used to move data into distributed data marts. This situation can arise in organizations where the data is stored centrally in a corporate data warehouse, and then needs to be delivered for the analysis to remote users at widely distributed locations. For example, the New York corporate headquarters of an international corporation collect critical data on various financial transactions into the enterprise wide data warehouse. The corporation has developed a robust multidimensional model to perform complex multivariate analysis using an OLAP tool. The financial analysts who benefit from this data reside in New York and in regional locations (London, Tokyo, Frankfurt). As soon as data becomes available to produce the required multidimensional database, this data is then securely replicated to all locations, loaded into local copies of the multidimensional data mart, and analyzed locally using the OLAP tool.

Database Gateways:

Aside from the high cost of development, many data warehouse solutions require the use of a database gateway. While traditional gate way technology provides LAN users with the ability to easily access small amounts of mainframe data, it is not optimized for moving large files. Net works can be slowed by multiple concurrent user requests for similar data through a gateway. Because gateway queries are not predictable, the DBMS cannot be tuned for gateway access. Gateway access for decision support will often compete with production applications for resources.

Metadata:

A frequently occurring problem in data warehousing is the problem of communicating to the end user what information resides in the data warehouse and how it can be accessed. The key to providing users and applications with a roadmap to the information stored in the warehouse is the metadata. It can define all data elements and their attributes, data sources and timing, and the rules that govern data use and data transformations. Metadata needs to be collected as the warehouse is designed and built. Since metadata describes the information in the warehouse from multiple viewpoints (input, sources, transformation, access, etc.) it is imperative that the same metadata or its consistent replicas be available to all tools selected for the warehouse implementation, thus enforcing the integrity and accuracy of the warehouse information. The metadata also has to be available to all warehouse users in order to guide them as they use the warehouse. Even though there are a number of tools available to help users understand and use the warehouse, these tools need to be carefully evaluated before any purchasing decision is made. In other words, a well thought through strategy for collecting, maintaining, and distributing metadata is needed for a successful data warehouse implementation. An interesting approach to metadata repository management is offered by Prism Directory Manager. This product is designed to integrate and manage all metadata definitions throughout the warehouse environment. The directory manager can:

Import business models from CASE tools (Bachman, ADW, IEF, and any other CDIF compliant CASE tools).

Import metadata definitions from the Prism Warehouse Manager (the transformation and extraction component of the Prism product suite).

Export metadata into catalogs, dictionaries, or directories of many DSS access tools.

Create flexible customized views based on end user requirements using graphical front end application.

In other words, Prism Directory Manager may be the right tool to solve the problem of creating and managing a unified metadata repository to support the entire data warehousing environment. A more detailed discussing on metal data management tools is offered in chapter.

User sophistication levels:

Data warehouse is a relatively new phenomenon, and a certain degree of sophistication is required on the end users part to effectively use the warehouse. A typical organization maintains different levels of computer literacy and sophistication within the user community. The users can be classified on the basis of their skill level in accessing the warehouse. For example, lets define three classes of users:

Casual users. These users are most comfortable retrieving information from the warehouse in predefined formats, and running preexisting queries and reports. These users do not need tools that allow for sophisticated ad hoc query building and execution.

Power users. In their daily activities, these users typically combine predefined queries with some relatively simple ad hoc queries that they create themselves. These users can also engage in drill down queries to further analyze the results of simple queries and reports. These users need access tools that combine the simplicity of predefined queries and reports with a certain degree of flexibility.

Experts. These users tend to create their own complex queries and perform a sophisticated analysis on the information they retrieve from the warehouse. These user known the data, tools, and database well enough to demand tools that allow for maximum flexibility and adaptability.

Therefore, when implementing the warehouse, an analysis of the end user requirements has to be coupled with an evaluation of the user sophistication levels, and the selected end user tools have to satisfy both the business needs and the capabilities and preferences of the end users.

Integrated solutions to build a Data warehouse:

Most data warehouse vendor solutions consist of a relational database used for the data warehouse, data warehouse management software, and data access and reporting tools, along with the necessary database connectivity software. What follows is a brief look at some of the solutions available on the market today.

A number of vendors participate in data warehousing by providing a suite of services and products that go beyond one particular component of the data warehouse. These vendors tend to establish internal centers of data warehousing expertise and often engage in partnership relationships with specialized independent software vendors for the products and expertise.

Digital Equipment Corp:

Digital has combined the data modeling, extraction, and cleansing capabilities of Prism Warehouse Manger with the copy management and data replication capabilities of Digitals ACCESSWORKS family of database access servers in providing users with the ability to build and use information warehouses. ACCESSWORKS runs on the Digital Open VMS platform and provides the necessary data access back end gateways and front end connections.

Hewlett Packard:

Hewlett Packards client / server based HP Open Warehouse comprises multiple components, including a data management architecture, the HP UX operating system, HP 9000 computers, warehouse management tools, an All base / SQL relational database, and the HP Information Access query tool. HP offers single source for the full Open Warehouse solution. This allows customers to choose components that best suit their needs, without having to work with multiple vendors for support. HP also provides a suite of consulting services for designing and implementing a data warehouse. HP Open Warehouse integrates a number of third party products, including Red Brick Warehouse from Red Brick Systems, Prism Warehouse Manager from Prism Solutions, EXTRACT from Evolutionary Technologies, Enterprise Data Access / SQL from IBI, and Open Development Environment from Open Environment Corp.

IBM:

The IBM Information Warehouse framework consists of an architecture; data management tools; OS /2, AIX, and MVS operating systems; hardware platforms, including mainframes and servers; and a relational DBMS (DB2). Other components of the IBM Information Warehouse family include.

Data Guide / 2 provides a catalog of shared data and information objects which can be listed and described in everyday business terms, and searched with key words. Once the information is identified, Data Guide/2 can start applications to retrieve and process it.

Data Propagator automates data replication across an enterprise, using a graphical user interface

Data Refresher provides copy management capability for moving large amounts of data from a broad range of data sources.

Data Hub manages complex client / server relational database environments and provides the base for IBMs family of copy management and replication tools.

Application System (AS) provides a set of general data access, query, reporting, and charting facilities and specialized tools for planning or modeling, project management, linear programming, document processing, and application development.

Personal Application System / 2 (PAS / 2) decision support software designed for windows and OS /2 PCs. The user can select from a list of decision support objects that can manage and analyze data, produce reports and charts of the data, and create procedures to automate regular work. Optionally includes business planning, project management, statistical analysis, and application development.

Query Management Facility (QMF) a host based facility that provides query, reporting, and graphics functions from menu options.

IBM Flow Mark workflow management product designed to help database administrators document their system management tasks and automatically execute these tasks.

Sequent:

Sequent Computer Systems Inc.s Decision Point Program is a decision support program for the delivery of data warehouse dedicated to on line complex query processing (OLCP). The program combines Sequent symmetric multiprocessing (SMP) architecture with a variety of client / server products and services, including UNIX based Sequent Symmetry 2000 Series, Red Brick Warehouse for Red Brick Systems, and Clear Access Query Tool form Clear Access Corp. Decision Point is targeted at information service organizations seeking to meet the data access demands of knowledge workers and executives. Clear Access Query Tool provides Decision Point users with a transparent window into the Red Brick Warehouse. Using a graphical interface, users query the data warehouse by pointing and clicking on the warehouse data items they want to analyze. Query results are placed on the programs clipboard for pasting onto a variety of desktop applications, or they can be saved to a disk.

Conclusion:

Clearly, this is not a complete list. Database vendors are all positioning themselves as providers of integrated data warehouse services and products. Oracle, Sybase, and Informix all offer integrated data warehouse solutions that are based on recent acquisitions, development of in house consulting expertise, and strategic partner alliances.

DATA EXTRACTION, CLEANUP, TRANSFORMATION AND MIGRATION

Tool Requirements:

The tools that enable sourcing of the proper data contents and formats from operational and external data stores into the data warehouse have to perform a number of important tasks that include

Data transformation from one format to anther on the basis of possible difference between the source and the target platforms.

Data transformation and calculation based on the application of the business rules that force certain transformations. Examples are calculating age from the data of birth or replacing a possible numeric gender code with a more meaningful male and female and data cleanup, repair, and enrichment, which may include corrections to the address field based on the value of the postal zip code.

Data consolidation and integration, which may include combining several source records into a single record to be loaded into the warehouse.

Metadata synchronization and management, which includes storing and / or updating metadata definitions about source data files, transformation actions, loading formats, and events, etc., thus, the entire data sourcing process is controlled by the documented in the metadata repository.

When implementing a data warehouse, several selection criteria that affect the tools ability to transform, consolidate, integrate, and repair the data should be considered.

The ability to identify data in the data source environments that can be read by the conversion tool is important. This additional step may affect the time lines of data delivery to the warehouse.

Support for flat files, indexed files (e.g., VSAM) and legacy DBMSs (e.g., IMS and CA IDMA) is critical, since the bulk of corporate data is still maintained in data stores of this type.

The capability to merge data from multiple data stores is required in many installations. Using data replication technologies helps a data change capture facility provides the functionality of reading log data to obtain only the changed data to be transferred. This reduces the amount of data that must be loaded into the data warehouse for periodic updating of the data warehouse.

The specification interface to indicate the data to be extracted and the conversion criteria is important.

The ability to read information form data dictionaries or import information from repository products is desired. This can reduce the amount of specification effort required.

The code generated by the tool should be completely maintainable from within the development environment.

Selective data extraction of both data elements and records enables users to extract only the required data.

A field level data examination for the transformation of data into information is needed. for example, a user exist feature is important for users who may need to perform more sophisticated operations on the data than can be achieved using the facilities provided with the tool. The ability to perform data type and character set translation is a requirement when moving data between incompatible systems. The capability to create summarization, aggregation, and derivation records and fields is very important. The data warehouse database management system should be able to perform the load directly from the tool, using the native API available with the RDBMS: alternatively, the system should support the capability to create a flat file and then use the load utility provided by the RDBMS. Vendor stability and support for the product are items that must be carefully evaluated.Vendor Approaches:

Prism Solutions:

While Enterprise / Access focuses on providing access to legacy data, Prism Warehouse Manager provides a comprehensive solution for data warehousing by mapping source data to a target database management system to be used as a warehouse. Warehouse Manager generates code to extract and integrate data, create and manage metadata, and build a subject oriented, historical base. The standard conversions, key changes, structural changes, and condensations needed to transform operational data into data warehouse information are automatically created. Prism Warehouse Manager can extract data from multiple source environments, including DBW, IDMS, IMS, VSAM, RMS, and sequential files under UNIX or MVS. Target databases include ORACLE, SYBASE, and INFORMIX. Prism solutions has strategic relationships with Pyramid and Informix. Prism Solution is discussed in more detail in the next chapter, which focuses on metadata.

Carleton Corporations PASSPORT and MetaCenter:

As was already mentioned, the process of migrating and moving legacy data to a new data warehouse environment can be costly and time consuming. And once it is built, managing the warehouse, from scheduling updates through metadata browsing, can be rather difficult and time consuming. Carletons PASSPORT and the MetaCenter products are positioned to fulfill these data extraction and transformation needs of data warehousing.

PASSPORT:

PASSPORT is a sophisticated metadata driven, data mapping, and data migration facility. PASSPORT Workbench runs as a client on various PC platforms in the three tiered environment, including OS/2 and Windows, and is used to develop PASSPORT applications to meet data warehousing and data migration needs.

The product currently consists of two components. The first, which is mainframe based, collects the file, record, or table layouts for the required inputs and outputs and converts them to the Passport Data Language (PDL). The second component is workstation based and is used to create the metadata directory from which it builds the COBOL programs to actually create the extracts. The user must transfer the PDL file from the mainframe to a location accessible by PASSPORT. The metadata directory is stored in a relational database (such as DB2/2). Carletons PASSPORT can produce multiple output files from a single execution of an extract program.

Overall, PASSPORT offers

A metadata dictionary at the core of the process

Robust data conversion, migration, analysis, and auditing facilities

The PASSPORT Workbench, a graphical user interface (GUI) workbench that enables project development on a workstation, with uploading of the generated application to the source data platform

Native interfaces to existing data files and RDBMS, helping users to lever age existing legacy applications and data

A comprehensive fourth generation specification language and the full power of COBOL

PASSPORT can assist the various personnel who design, implement, maintain, or use data warehouses, including business systems analysts and those involved in joint application design (JAD) sessions, data warehouse designers and implementors, data warehouse administrators, and end users.

PASSPORT highlights. Carleton PASSPORT includes a number of facilities and features which are briefly discussed below.

Data Access. This facility is data dictionary driven, and provides input data selection, automatic file and data matching, selective random access, automatic handling of multiple record types, intelligent joins, data sampling, and single or multiple record access.

Data Analysis and Auditing. This facility provides

Audit reports, including Duplicates report, Sequence Checking report, Cross Tabulation report, File Footing report, Stratification report, and Statistical Summary report.

Audit Facilities, including SAMPLE Command, AGE Function, DATE Keyword, and ENCRYPT Option.

Language and Design. This facility supports predefined calculations, arithmetic operations, relational and Boolean operations, range operations, arrays, input data sorts, work fields and external subroutines, and loop processing.

Passport Data Language (PDL). This has a free from command structure, with English like command syntax.

Run time environment. This supports dynamic work fields and error limit control.

Report Writing. This facility supports an unlimited number of line formats, unrestricted output line contents, variable page size, controlled horizontal and vertical spacing, and dynamic printing.

Additionally, PASSPORT provides the System Activity Report, which includes syntax checking, error trapping, run time messages, file activity summary, report activity summary, and path analysis.

Other features of Carleton PASSPORT include

Centralized metadata repository. This is an enterprise level, common, single master metadata repository that provides global access, provides central information change management and control, and enables metadata accuracy and integrity.

Business metadata. Carletons PASSPORT provides for metadata in business English in addition to a technical description for end users to access. This business metadata is stored in a relational format that is accessible by any SQL based query tool.

Load image formats for target DBMS. PASSPORT formats data for lading into any target RDBMS, including DB2, Informix, Oracle, Sybase, and Red Brick. For example, PASSPORTs DB2 internal load file format provides for very efficient loading into DB2, several times faster than the DB2 external load format.

Optional user exits. PASSPORT provides support for user exits, where users can optionally invoke previously written routines.

Browsing capabilities. PASSPORT provides metadata in a relational format that is easily accessible by any end user query tool. Metadata is collected and stored in a format that is optimized to support a wide range of browsing and querying requirements.

DBMS SCHEMAS FOR DECISION SUPPORTStar Schema:

The multidimensional view of data that is expressed using relational database semantics is provided by the database schema design called star schema. The basic premise of star schemas is that information can be classified into two groups: facts and dimensions. Facts are the core data element being analyzed. For example, units of individual items sold are facts, while dimensions are attributes about the facts. For example, dimensions are the product types purchased and the data of purchase.

Asking the business question against this schema is much more straight forward because we are looking up specific facts (UNITS) through a set of dimensions (MARKETS, PRODUCTS, PERIOD). Its important to notice that, in the typical start schema, the fact table is much larger than any of its dimension tables. This point becomes an important consideration of the performance issues associated with star schemas.

Consider a typical business analysis problem: Find the share of total sales represented by each product in different markets, categories, and periods, compared with the same period a year ago. To do so, you would calculate the percentage each number is of the total of its column, a simple and common concept.

Facts Table primary key = {timekey:prodkey:mktkey}

Figure: Star schema

However, in a classic relational database these calculations and display would require definition of a separate view, requiring over 20 SQL commands. The star schema is designed to overcome this limitation of the two dimensional relational model.

DBA viewpoint:

A start schema is a relational schema organized around a central table (fact table) joined to a few smaller table (dimensional tables) using foreign key references. The fact table contains raw numeric items that represent relevant business facts (price, discount values, number of units sold, dollar value, etc.). The facts are typically additive and are accessed via dimensions. Since the face tables are presummarized and aggregated along business dimensions, these tables tend to be very large.

Smaller tables are the dimensional tables, which define business dimension in terms already familiar to the users. These dimensional tables contain a non compound primary key (mktkey, timekey, etc.) and are heavily indexed. Dimensional tables typically represent the majority of the data elements. These tables appear in constraints and GROUP BY clauses, and are joined to the fact table using foreign key references. Typical dimensions are time periods, geographic and demographic regions, products, promotions and discounts, sales representatives, office numbers, account numbers, vendors, distributors, etc.

Once the star schema database is defined and loaded, the queries that answer questions similar to the one described here are relatively simple, and, whats more, the actual database work to arrive at the answer is minimal compared to a standard relational model.

A star schema can be created for every industry consumer packaged goods, retail, telecommunications, transportation, insurance, health care, manufacturing, banking, etc.

Potential performance problems with star schemas:

Although most experts agree that star schemas are the preferred modeling method for data warehousing, there are a number of RDBMS issues associated with star schema implementation. In general, if a query centric star schema is hosted on a traditional RDBMS optimized for OLTP processing, the result may be poor performance.

METADATA

Definition:

Metadata is one of the