www.synerzip.com

Microsoft Business Intelligence (MSBI),

Data Warehousing (DW) andData Integration Techniques

Presenter,Valmik Potbhare

http://lnkd.in/bp_3eFm

1

May 21, 2014

Agenda

Confidential

What is BI?

What is Data Warehousing?

Microsoft platform for BI applications

Data integration methods

T-SQL examples on data integration

2

What is BI?

3Confidential

Business Intelligence is a collection of theories,

algorithms, architectures, and technologies that

transforms the raw data into the meaningful data in

order to help users in strategic decision making in

the interest of their business.

BI Case

4Confidential

For example senior management of an industry

can inspect sales revenue by products and/or

departments, or by associated costs and incomes.

BI technologies provide historical, current and

predictive views of business operations. So,

management can take some strategic or operation

decision easily.

Typical BI Flow

5Confidential

Data Sources

Users

Data Tools

Data Warehouse

Extraction

Why BI?

6Confidential

By using BI, management can monitor objectives from high level, understand what is happening, why is happening and can take necessary steps why the objectives are not full filled.

Objectives:1)Business Operations Reporting2)Forecasting3)Dashboard4)Multidimensional Analysis5)Finding correlation among different factors

What is Data warehousing?

7Confidential

A data warehouse is a subject-oriented, integrated, time-variant and non-volatile collection of data in support of management's decision making process. - Bill Inmon

A data warehouse is a copy of transaction data specifically structured for query and analysis.- Ralph Kimball

Dimensional Data Model

8Confidential

Although it is a relational model but data would be stored differently in dimensional data model when compared to 3rd normal form.

Dimension: A category of information. Ex. the time dimension.Attribute: A unique level within a dimension. Ex. Month is an attribute in the Time Dimension.Hierarchy: The specification of levels that represents relationship between different attributes within a dimension. Ex. one possible hierarchy in the Time dimension is Year → Quarter → Month → Day.Fact Table: A fact table is a table that contains the measures of interest. Ex. Sales Amount is a measure.

Data warehouse designs

9

• Star Schema – A single object (the fact table) sits in the middle and is radically connected to other surrounding objects (dimension lookup tables) like a star. Each dimension is represented as a single table. The primary key in each dimension table is related to a foreign key in the fact table.• Snowflake Schema – An extension of the star schema, where each point of the star explodes into more points. In a star schema, each dimension is represented by a single dimensional table, whereas in a snowflake schema, that dimensional table is normalized into multiple lookup tables, each representing a level in the dimensional hierarchy.

Confidential

Typical Data warehouse model

10Confidential

Data warehouse implementation

11

After the team and tools are finalized, the process follows below steps in waterfall:a)Requirement Gathering

b)Physical Environment Setup

c)Data Modeling

d)ETL

e)OLAP Cube Design

f)Front End Development

g)Report Development

h)Performance Tuning and Query Optimization

i)Data Quality Assurance

j)Rolling out to Production

k)Production Maintenance

l)Incremental Enhancements

Confidential

Microsoft BI Platform

12Confidential

Microsoft BI Tools

Confidential

SSIS – This tool in MSBI suite performs any kind of data transfer with flexibility of customized dataflow. Used typically to accomplish ETL processes in Data warehouses.

SSRS – provides the variety of reports and the capability of delivering reports in multiple formats. Ability to interact with different kind of data sources

SSAS – MS BI Tool for creating a cubes, data mining models from DW. A typical Cube uses DW as data source and build a multidimensional database on top of it.

13

MSBI Tools

14

Power View and Power Pivot – These are self serve BI tools provided by Microsoft. Very low on cost of maintenance and are tightly coupled with Microsoft Excel reporting which makes it easier to interact.

Performance Point Servers – It provides rapid creation of PPS reports which could be in any form and at the same time forms can be changed just by right click.

Microsoft also provides the Scorecards, dashboards, data mining extensions, SharePoint portals etc. to serve the BI applications.

Confidential

Confidential

Data Integration methods

15

Different ways of integration

16

RDBMS – •Copying data from one table to another table(s)•Bulk / Raw Insert operations•Command line utilities for data manipulation•Partitioning data

File System – •Copying file(s) from one location to another•Creating flat files, CSVs, XMLs, Excel spreadsheets•Creating directories / sub-directories

Confidential

Different ways of integration

17

Web –

•Calling a web service to fetch / trigger data

•Accessing ftp file system

•Submitting a feedback over internet

•Sending an email / SMS message

Other –

•Generate Auditing / Logging data

•Utilizing / maintaining configuration data (static)

Confidential

Confidential

T-SQL Best practices

18

Query to merge data into a table

Confidential 19

MERGE dbo.myDestinationTable AS dest

USING (

SELECT ProductID

, MIN(PurchaseDate) AS MinTrxDate

, MAX(PurchaseDate) AS MaxTrxDate

FROM dbo.mySourceTable

WHERE ProductID IS NOT NULL

GROUP BY ProductID

) AS src

ON dest.ProductID = src.ProductID

WHEN MATCHED THEN

UPDATE SET MaxTrxDate = src.MaxTrxDate

, MinTrxDate = ISNULL(dest.MinTrxDate, src.MinTrxDate)

WHEN NOT MATCHED BY SOURCE THEN DELETE

WHEN NOT MATCHED BY TARGET THEN INSERT (ProductID, MinTrxDate, MaxTrxDate)

VALUES (src.ProductID, src.MinTrxDate, src.MaxTrxDate);

MERGE clause is T-SQL programmers’ favorite as it covers 3 operations in MERGE clause is T-SQL programmers’ favorite as it covers 3 operations in oneone

Query to get a sequence using CTE

Confidential 20

;WITH myTable (id) AS

(

SELECT 1 id

UNION ALL

SELECT id + 1 FROM myTable

WHERE id < 10

)

SELECT * FROM myTable

COMMON TABLE EXPRESSIONS (CTEs) are the most popular COMMON TABLE EXPRESSIONS (CTEs) are the most popular recursive constructs in T-SQLrecursive constructs in T-SQL

Move Rows in a single Query

Confidential 21

DECLARE @Table1 TABLE (id int, name varchar(50))

INSERT @Table1 VALUES (1, 'Maxwell'), (2, 'Miller'), (3, 'Dhoni')

DECLARE @Table2 TABLE (id int, name varchar(50))

DELETE FROM @Table1 OUTPUT deleted.* INTO @Table2

SELECT * FROM @Table1

SELECT * FROM @Table2

OUTPUT clause redirects the intermediate results of OUTPUT clause redirects the intermediate results of UPDATE, DELETE or INSERT into a table specifiedUPDATE, DELETE or INSERT into a table specified

Query to generate random password

Confidential 22

SELECT CHAR(32 + (RAND() * 94))

+CHAR(32 + (RAND() * 94))

+CHAR(32 + (RAND() * 94))

+CHAR(32 + (RAND() * 94))

+CHAR(32 + (RAND() * 94))

+CHAR(32 + (RAND() * 94))

Non-deterministic functions like RAND() gives Non-deterministic functions like RAND() gives different result for each evaluationdifferent result for each evaluation

Funny T-SQL – Try it yourself

Confidential 23

Aliases behavior is not consistent

SELECT 1id, 1.eMail, 1.0eMail, 1eMail

Ever seen WHERE clause in SELECT without FROM clause ?

SELECT 1 AS id WHERE 1 = 1

IN clause expects column name at its left? Well, not Really!

SELECT * FROM myTable WHERE 'searchtext' IN (Col1, Col2, Col3)

Two ‘=‘ operators in single assignment in UPDATE? Possible!

DECLARE @ID INT = 0

UPDATE mySequenceTable SET @ID = ID = @ID + 1

Confidential 24

Contact Us:

contact@synerzip.com