data intelligence overview

Data, Data Warehouse and Business Intelligence

DWM

Data Intelligence

Copyright DWM Consulting 2014


• Getting data from multiple systems

• Defining and implementing standards

• Formatting to a generally accepted standard

• Storage of a huge volume of information

• Distribution to humans and other systems

• Usability by humans and other systems

What is Data Intelligence all about?


A bird’s eye view of the entire process for human use

Data and Intelligence Services

This diagram is a simplified presentation of a process that is designed to create order out of disorder and present end users with information that can be used to more effectively run an Enterprise.

Data source Homogeniser Storage Distribution Use

Frequently referred to as a “legacy“ system.

However, a data source can be anything from an old system to a social network.

Frequently referred to as “ETL“.

The homogeniser transforms data disorder into data order so that apples are apples and oranges are oranges.

Frequently referred to as a “Data Warehouse“.

This is where all the homogenised data is stored. All data conforms to a single standard.

Frequently referred to as “Data Marts“.

These are subsets of the Data Warehouse information that are designed for a specific business line.

Frequently referred to as “Analytical Engines“.

This is the process where end users can query information from the data marts they use.


A bird’s eye view of the entire process for computer use


This diagram is a simplified presentation of a process that is designed to create order out of disorder and present enterprise systems with homogenised information to avoid confusion and prevent failure.

Data source Homogeniser Storage Distribution System

Frequently referred to as a “legacy“ system.

However, a data source can be anything from an old system to a social network.

Frequently referred to as “ETL“.

The homogeniser transforms data disorder into data order so that apples are apples and oranges are oranges.

Frequently referred to as a “Data Warehouse“.

This is where all the homogenised data is stored. All data conforms to a single standard.

This differs from Data Marts because the distribution system will send information to another computer system.

Another system that needs homogenised data.

It could be the GL, client information, risk management, EDI, social networks, etc.


• A system that will provide data for data intelligence – Enterprise systems: Internal regardless of age.– External systems: market data, EDI, Swift, Bloomberg, etc.– Social networks: Facebook, Twitter, LinkedIn, etc.

Data source



• A system that creates data order by:– Standardising formats– Standardising business meaning e.g. a financial amount– Standardising reference data codes e.g. client id, currency code, etc.– Standardising operations e.g. orders, sales, etc.– Adding supplementary or missing information– Report errors and exceptions during the process– Usual package choices IBM, Informatica, Microsoft, Oracle

Homogeniser



• A central data base of ordered data– AKA Data Warehouse– Coherent, consistent and standardised information– A “single version of the truth” – Bill Inmon– Usually a VLDB (Very Large Data Base)– Usually a relational data base using SQL (Structured Query Language)– Usual VLDB choices: Greenplum, IBM, Microsoft, Oracle, Teradata – The information may be spread across several inter-connected data bases

but they are perceived as a single unit

Storage



• Creates Data Marts (subsets) of the Data Warehouse– Extracts information from the core Data Warehouse using predefined rules– Creates/updates a user group’s information source (Data Marts)– Report errors and exceptions during the process– Owing to the smaller size of the Data Marts in relation to the core Data

Warehouse, reports and queries usually run faster– Each Data Mart is designed for a specific business purpose whereas the

core Data Warehouse is general purpose– Usually a relational data base using SQL (Structured Query Language)– Usual data base choices: Greenplum, IBM, Microsoft, Oracle, Terradata

Distribution for human use



• Sends information from the Data Warehouse to other systems– Extracts information from the core Data Warehouse using predefined rules– Report errors and exceptions during the process– Sends the information using a predefined protocol to other systems– Data sending will depend on the target system’s interface– System interfaces are set by the manufacturer

Distribution for computer use



• Use the data for running the business– Allows people to query the data and create reports graphically or in print – Data can be presented graphically (visualised) or in character format– Allows people to create their own reports that can then be saved for re-

use and/or distribution– Usually does not allow data in the Data Mart to be changed– Usually uses specialised query and presentation software– Common software choices are Business Objects, Cognos, Hyperion, Qlik ,

SAS. There are many other choices for this type of software.

Use by humans



• Use the data for other purposes– Data sent to other systems will be used for whatever purpose each system

needs– The individual purpose of these systems is outside the scope of this

presentation

Use by other systems



• We will explain the best way to organise a Data Intelligence project

• We will examine each step

• We will highlight what needs to be done

• We will highlight the required resources

• We will explain the pitfalls and problems that may arise

• We will provide workable solutions

Moving on from here



• Phone: (+34) 626 341 273 or (+34) 915 553 975

• Collaborations with– Bishopsgate Financial, the City, London– Orange Grove Inc., Santa Monica, California

Contact and collaborations


data intelligence overview

Data & Analytics

systems data

data order

market data

print data

thecore data warehouse

relational data base

large data base

data marts subsets