keeping it tidy- how to warehouse your sharepoint data
DESCRIPTION
We all know that SharePoint is an excellent tool for gathering information and collaborating. What it is less good at it maintaining large quantities of that information over long periods of time. The good news is that SQL Server is very good at exactly that. In the spirit of choosing the right tool for the right job, this session will present a methodology for moving your historical SharePoint information from SharePoint and into a SQL Server based data warehouse. From there, we will cover several method for then surfacing data back in SharePoint in context.TRANSCRIPT
John WhiteChief Technology OfficerUnlimitedViz Inchttp://whitepages.unlimitedviz.com@diverdown1964
How to Warehouse Your SharePoint Data
Keeping It Tidy
2 | SharePointalooza – Branson, MO 2014
A Few Reminders
Download the attendee packet at http://bit.ly/SPAloozaAttendee
Attend the “Rock Star” Sessions at the end of each day for fun, raffle prizes, wrist bands for concert access, and your chance to win a Surface Pro 3
Attend Nintex’s Brown Bag lunch Friday and Saturday (lunch provided for first 100 people)
Tweet about the event using #SharePointalooza Thank our sponsors Have a great time!
3 | SharePointalooza – Branson, MO 2014
Thank you, sponsors!Rock Star Sponsors
Performer Sponsors
Giveaway Sponsors
Soloist Sponsors
4 | SharePointalooza – Branson, MO 2014
The Bands
What better way to unwind after a long day of working out your brain than with some great live music at the amazing outdoor stage at Branson Landing! The bands will be playing both Friday and Saturday night from 6:30 pm to 10 pm.
John WhiteCTO/Co-Founder of UnlimitedVizSharePoint Server MVP, SQL Server v-TS
[email protected]://whitepages.unlimitedviz.com@diverdown1964
6 | SharePointalooza – Branson, MO 2014
Agenda
SharePoint Data Storage Data Warehousing Moving SharePoint Data Surfacing Warehoused Data in SharePoint Q&A
The Problem
SharePoint Data Storage
8 | SharePointalooza – Branson, MO 2014
Excel Data Storage
List based data Powerful Features Massive adoption Good usability
NOT a database Not relational Capacity limits Not authoritative
Excellent BI Client Surface the data.. DON’T store it
SharePoint Content Databases
10 | SharePointalooza – Branson, MO 2014
SharePoint Content Databases (2010)
~160 fields
11 | SharePointalooza – Branson, MO 2014
SharePoint Capacity Boundaries
Boundaries and Limits - http://technet.microsoft.com/en-us/library/cc262787
200 GB per content DB (general usage scenarios - or up to 4 TB)
List view threshold – 5,000 simultaneous items
List view lookup threshold – 8 join operations per query
Items per content database – 60 million
12 | SharePointalooza – Branson, MO 2014
WorkflowHistory List
Demo
Data Warehousing
ONE DOES NOT SIMPLY
RETURN HIS RAW DATA FROM THE DATABASE
15 | SharePointalooza – Branson, MO 2014
BI Architecture 101
Data Marts
Extract, Transform, and Load (ETL)
Middleware Server(s)
DataWarehouse
Storage Design and Visualization
Data Cubes and Tabular
Models
E
T
L
Reporting Server(s)
BI and Designer Clients
Source data
16 | SharePointalooza – Branson, MO 2014
EE
Microsoft enterprise (classic) BI
SQL Server DBSQL Server Integration Services (SSIS)
SharePoint (with)• Excel Services• PowerPivot for
SharePoint• SSRS SharePoint Mode• PerformancePoint
SQL Server DB
Storage Design and Visualization
SQL Server Analysis Services
Multidimensional and Tabular modes
L
SQL Server Reporting Services (SSRS)
ExcelSQL Data ToolsReport Builder3rd party tools
ETL
E
T
Source data
17 | SharePointalooza – Branson, MO 2014
Team BI and SharePoint Dashboards
Power Pivot Worksheets• Pivot Tables and
Charts• Power View
Data Marts and other
Data Cubes and Tabular
Models
Standard Worksheets• Pivot Tables and
Charts
PerformancePoint Reports• Analytic Charts and Grids• Decomposition trees
SQL Server Reporting Services Reports• Standard• Power View
PerformancePoint Scorecards and KPIs
18 | SharePointalooza – Branson, MO 2014
Why Warehouse?
Optimize for End Goal Multidimensional Flattened
Usability Reduced Load on Source Systems Security Requirements Licensing Requirements SharePoint (Performance)
Moving SharePoint Data
20 | SharePointalooza – Branson, MO 2014
ETL Tools
SQL Server Integration Services Comes with ALL Editions of SQL Server (except Express)
SQL Server Data Tools – BI Edition http://go.microsoft.com/fwlink/?LinkID=320744 (For VS2010) http://www.microsoft.com/en-ca/download/details.aspx?id=36843 (For VS2012) http://www.microsoft.com/en-us/download/details.aspx?id=42313 (For VS2013)
SharePoint List Source and Destination (CodePlex Project) http://sqlsrvintegrationsrv.codeplex.com/releases/view/17652 (Download) http://
dataqueen.unlimitedviz.com/2011/06/how-to-use-a-sharepoint-list-as-a-data-source-in-your-ssis-package/ (How to)
OData Source for Microsoft SQL Server 2012 (Official) http://www.microsoft.com/en-us/download/details.aspx?id=42280 (Download) http://whitepages.unlimitedviz.com/2014/03/using-the-odata-source-connector-with-sharepoint-on
line-authentication/ (How to)
SharePoint Server 2013 Client Components SDK - http://www.microsoft.com/en-us/download/details.aspx?id=35585 – Required for Office 365 authentication
21 | SharePointalooza – Branson, MO 2014
Moving SharePoint Data
Demo
Accessing Warehoused Data
23 | SharePointalooza – Branson, MO 2014
Data Consumption
Business Connectivity Services External List All SharePoint Editions (yes..Foundation)
SQL Server Reporting Services Advanced Reporting All Editions of SQL Server (even Express!) ..except Express doesn’t do SP Integrated Mode
Excel/Power Pivot Excel Services requires Enterprise SharePoint PowerPivot for SharePoint requires SQL Server Standard or better
24 | SharePointalooza – Branson, MO 2014
Accessing Warehoused Data
URL to report: http://home.nautilusdemo.com/sites/nmarine/finance/_layouts/15/ReportServer/RSViewerPage.aspx?rv:RelativeReportUrl=/sites/nmarine/finance/SSRSReports/SPAlooza.rdl&rv:ParamMode=Hidden&rp:ItemID=itemid, Our Workflow History
Demo
25 | SharePointalooza – Branson, MO 2014
Considerations
Transformations Mashups Scheduling Data latency
26 | SharePointalooza – Branson, MO 2014
In Conclusion
Querying SharePoint lists is bad for everyone Tools exist to move relevant data Warehousing applies to SharePoint too Easy to surface data in SharePoint
27 | SharePointalooza – Branson, MO 2014
Q&A
[email protected]@diverdown1964whitepages.unlimitedviz.com