sql server 2016 ctp technical deep dive

SQL Server 2016New innovations

There’s an opportunity to drive smarter decisions with data

The explosion of data sources...

…drives an explosion of data

…which drives businesses to learn more

2013-2020 CAGR = 41%

25B

4.0B

1.3B2010 2013 2020

Source: Forecast: Internet of Things, Endpoints and Associated Services, Worldwide, 2014. Gartner. Oct 20 2014Source: IDC “Digital Universe”, Dec. 2012

$ 1.6TData dividendEmployee

productivityOperations improvementProduct innovations

Higher sales

More people

New analytics

Diverse data

Speed

Source: IDC & Microsoft, April 2014

Data differentiates today's leading companies

Operational Database Management Systems

Data Warehouse Database Management Systems

Business Intelligence and Analytics Platforms

x86 Server Virtualization

Cloud Infrastructure as a Service

Enterprise Application Platform as a Service

Public Cloud Storage

Leader in 2014 for Gartner Magic QuadrantsMicrosoft platform leads the way on-premises and cloud

Hyperscale cloud

Deeper insights across data

Do more. Achieve more.

Performance Security Availability Scalability

Operational analyticsInsights on operational data; Works with in-memory OLTP and disk-based OLTP

In-memory OLTP enhancementsGreater T-SQL surface area, terabytes of memory supported, and greater number of parallel CPUs

Query data store Monitor and optimize query plans

Native JSON Expanded support for JSON data

Temporal database supportQuery data as points in time

Always encryptedSensitive data remains encrypted at all times with ability to query

Row-level securityApply fine-grained access control to table rows

Dynamic data maskingReal-time obfuscation of data to prevent unauthorized access

Other enhancementsAudit success/failure of database operations

TDE support for storage of in-memory OLTP tables

Enhanced auditing for OLTP with ability to track history of record changes

Enhanced AlwaysOnThree synchronous replicas for auto failover across domains

Round robin load balancing of replicas

Automatic failover based on database health

DTC for transactional integrity across database instances with AlwaysOn

Support for SSIS with AlwaysOn

Enhanced database caching Cache data with automatic, multiple TempDB files per instance in multi-core environments

Mission-critical performance

Operational AnalyticsWhat is operational analytics and what does it mean to you?

Operational analytics with disk-based tables

Operational analytics with In-Memory OLTP

BI and analytics

Add analytics-specific indexes

SQL ServerAnalysis Server

SQL Server

Database

Application tier

Presentation layer

IIS Server

BI analysts

Traditional operational/analytics architecture Key issues

Complex implementation

Requires two servers (capital expenditures and operational expenditures)

Data latency in analytics

More businesses demand; requires real-time analytics

BI and analytics

SQL Server

Database

Application tier

Presentation layer

IIS Server

ETL

Hourly, Daily, Weekly Database

SQL ServerRelational DW


BI analysts

Minimizing data latency for analytics BenefitsNo data latency

No ETL

No separate data warehouse

ChallengesAnalytics queries are resource intensive and can cause blocking

Minimizing impact on operational workloads

Sub-optimal execution of analytics on relational schema

BI and analytics

Add analytics-specific indexes


SQL Server

Database

Application tier

Presentation layer

IIS Server

BI analysts

Operational analytics forin-memory tables

Operational analytics: Columnstore on in-

memory tables SQL Server 2016 – CTP2 limitationYou can create columnstore index on empty tableAll columns must be included in the columnstore

No explicit delta rowgroupRows (tail) not in columnstore stay in in-memory OLTP tableNo columnstore index overhead when operating on tailBackground task migrates rows from tail to columnstore in chunks of 1 million rows not changed in last 1 hour.

Deleted Rows Table (DRT) – Tracks deleted rows

Columnstore data fully resident in memory Persisted together with operational data

No application changes required.

In-Memory OLTP table

Updateable CCI

TailDRT

Range Index

Hash index

Hot

Like Delta rowgroup

Operational Analytics: Columnstore Overhead

Operation Hash or Range Index HK-CCI

Insert Insert row into HK Insert row into HK

Delete (a)Seek row(s) to be deleted

(b)Delete the row

(a)Seek row(s) to be deleted(b)Delete the row in HK(c) If row in TAIL then return else insert <colstore-RID> into DRT

Update (a)Seek the row(s) (b)Update (delete/insert)

(a)Seek the row(s) (b)Update (delete/insert) in HK(c) If row in TAIL then return else insert <colstore-RID> into DRT

DML operations on In-Memory OLTP

Operational Analytics: Minimizing Columnstore

overhead DML OperationsKeep hot data only in-memory tablesExample – data stays hot for 1 day, 1 week…

CTP2: Work-AroundUse TF – 9975 to disable auto-compressionForce compression using a spec-proc“sp_memory_optimized_cs_migration”

Analytics queriesOffload analytics to AlwaysOn Readable secondary

In-Memory OLTP table

Updateable CCI

TailDRT

Range Index

Hash index

Hot

Like Delta rowgroup

Summary: Operational analyticsData Warehouse queries can be run on in-memory OLTP workload with no application changes.

These operations have minimal impact on OLTP workload.

Best performance and scalability available

Offloading analytics workload to Readable Secondary

Minimizing impact on OLTP Example: for HOT/WARM (Predicate)

Minimizing impact on OLTP Example: for HOT/WARM (Time Based)

Operational analytics disk-based tables

Columnstore feature summaryFunctionality:Fast equality search and short-range scans

PK/FK enforcement

Coarser concurrencyLocking granularity: Impact MERGE load or UPDATES

No support of SI/RCSI

Full support of MARS (no updates)

Improved supportability

Performance and high availabilityOnline index maintenance

Batch operations: SORT

Window Aggregates

Single

Threaded Execution

String predicate push down

Single Instruction Multiple Destination (SIMD) support

Aggregate Pushdown

Parallel Insert

Loading data is very primitive

Query offload to AlwaysOn Readable Secondaries

Operational analytics with columnstore index Key points

Create an updateable NCCI for analytics queries

Drop all other indexes that were created for analytics

No application changes

Columnstore index is maintained just like any other index

Query Optimizer will choose columnstore index where needed

Row Group 1

Row Group 2

Row Group 3

Columns A B C

Dele

te B

itm

ap

Delta Table Row

Oriented

Modifiacations of Rows Clusters Column Store

Online Merge

Insert of Rows

Operational analytics: Columnstore index

overhead

Operation B-tree (NCI) Non-clustered columnstore index (NCCI)

Insert Insert row into b-tree. Insert row into b-tree (delta store).

Delete (a)Seek row(s) to be deleted.

(b)Delete the row.

(a)Seek row in delta stores. (There can be multiple rows.(b)If found, delete row.(c) If not found, insert key into delete row

buffer.

Update (a)Seek the row(s) .(b)Update.

(a)Delete row (steps same as above).(b)Insert updated row into delta store

Operational analytics with columnstore indexDML operations on OLTP workload

--Create the tableCREATE TABLE t_account ( accountkey int PRIMARY KEY, accountdescription nvarchar (50), accounttype nvarchar(50), unitsold int);

--Create the columnstore index with a filtered conditionCREATE NONCLUSTERED COLUMNSTORE INDEX account_NCCI ON T_account (accountkey, accountdescription, unitsold) WHERE accountkey > 0;

--For this SELECT statement, the query optimizer will consider using the --columnstore index for the query plan, in which case it will also retrieve --all rows with accountkey = 0 from the rowstore table.SELECT SUM( unitsold ) FROM t_account WHERE accountkey >=0 AND accountkey < 1000;

Access hot data from btree index, warm data from columnstore index

This example uses a filtered condition (accountkey > 0) to establish which rows will be in the columnstore index

When the query optimizer chooses the filtered columnstore index, the query will return the correct results even when the results contain rows that do not meet the filtered condition. This is different from a regular nonclustered filtered index which can be used only in queries that restrict themselves to the rows present in the index.

Minimizing columnstore overhead for DML

Using Availability Groups instead of data warehouse Key points

Mission Critical Operational Workloads typically configured for High Availability using AlwaysOn Availability Groups

You can offload analytics to readable secondary replica

Secondary Replica

Secondary Replica

Secondary Replica

Primary Replica

Always on Availability Group

Summary of improvementsImprovements SQL Server 2014 SQL Server 2016

clustered columnstore index

Master copy of the data (10x compression)Only index supported; simplified analyticsNo PK/FK constraintsUniqueness can be enforced through Materialized ViewsLocking granularity for UPDATE/DELETE at rowgroup levelDDL: ALTER, REBUILD, REORGANIZE

Master copy of the data (10x compression)

Additional b-tree indexes for efficient equality, short-range searches,

and PK/FK constraints.

Locking granularity at row level using NCI index path

DDL: ALTER, REBUILD, REORGANIZE

updateable non-clustered index

Introduced in SQL Server 2012NCCI is read-only: no delete bitmap or delta storeOptimizer will choose between NCCI and NCI(s)/CI or heap-based on the cost-based modelPartitioning supported

Updateable

Ability to mix OLTP and analytics workload

Ability to create filtered NCCI

Partitioning supported

equality and short-range queries

Rowgroup elimination (when possible)

Partition-level scan (somewhat expensive)

Full index scan (expensive)

Optimizer can choose NCI on column C1; index points directly to

rowgroup

No full index scan

Covering NCI index

string predicate pushdown

Retrieve 10 million rows by converting dictionary encoded value to string

Apply string predicate on 10 million rows

Apply filter on dictionary entriesFind rows that refer to dictionary entries that qualify (R1)Find rows not eligible for this optimization (R2)Scan returns (R1 + R2) rowsFilter node applies string predicate on (R2)Row returned by Filter node = (R1 + R2’)

Support for Index MaintenanceOperation SQL Server 2014 SQL Server 2016

Removing deleted rows Requires index REBUILD Index REORGANIZERemove deleted rows from single compressed RGMerge one or more compressed RGs with deleted rowsDone ONLINE

Smaller RG size resulting fromSmaller BATCHSIZEMemory pressureIndex build residual

Index REBUILD Index REORGANIZE

Ordering rows Create clustered indexCreate columnstore index by dropping clustered index

No changes in CTP2

Query Rowgrou GranularityNo Support of RCSI or SIRecommendation: Use Read Uncommitted

Support of SI and RCSI (non blocking)

Insert Lock at row level (trickle insert)Rowgroup level for set of rows

No changes

Delete Lock at Rowgroup level Row level lock in conjunction with NCI

Update Lock at Rowgroup level Implemented as Delete/Insert

Row level lock in conjunction with NCI

AlwaysOn Failover Clustering (FCI) Fully supported Fully supported

AlwaysON Availability Groups Fully supported except readable secondary Fully supported with readable secondary

Index create/rebuild Offline Offline

In-memory OLTP enhancements

In-memory OLTP enhancements

ALTER TABLE Sales.SalesOrderDetail ALTER INDEX PK_SalesOrderID REBUILD WITH (BUCKET_COUNT=100000000)

T-SQL surface area: New

{LEFT|RIGHT} OUTER JOINDisjunction (OR, NOT)UNION [ALL]SELECT DISTINCTSubqueries (EXISTS, IN, scalar)

ALTER support

Full schema change support: add/alter/drop column/constraint

Add/drop index supported

Surface area improvements

Almost full T-SQL coverage including scaler user-defined functions

Improved scaling

Increased size allowed for durable tables; more sockets

Other improvements

MARS support

Lightweight migration reports

| ADD { <column_definition> | <table_constraint> | <table_index> } [ ,...n ]

| DROP { [ CONSTRAINT ] { constraint_name } [ ,...n ] | COLUMN { column_name } [ ,...n ] | INDEX { index_name } [ ,...n ] } [ ,...n ]

| ALTER INDEX index_name { REBUILD (WITH <rebuild_index_option>) }}

The ALTER TABLE syntax is used for making changes to the table schema, as well as for adding, deleting, and rebuilding indexes

Indexes are considered part of the table definition

Key advantage is the ability to change the BUCKET_COUNT with an ALTER INDEX statement

Altering memory-optimized tables

CREATE PROCEDURE [dbo].[usp_1]WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNERAS BEGIN ATOMIC WITH( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english') SELECT c1, c2 from dbo.T1ENDGO

ALTER PROCEDURE [dbo].[usp_1]WITH NATIVE_COMPILATION, SCHEMABINDING, EXECUTE AS OWNERAS BEGIN ATOMIC WITH( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N'us_english') SELECT c1 from dbo.T1ENDGO

You can now perform ALTER operations on natively compiled stored procedures using the ALTER PROCEDURE statement

Use sp_recompile to recompile stored procedures on the next execution

Altering natively compiled stored procedures

• CREATE PROCEDURE (Transact-SQL) • DROP PROCEDURE (Transact-SQL) • ALTER PROCEDURE (Transact-SQL) • SELECT (Transact-SQL) and INSERT SELECT statements• SCHEMABINDING and BEGIN ATOMIC (required for natively compiled stored procedures)• NATIVE_COMPILATION• Parameters and variables can be declared as NOT NULL • Table-valued parameters.• EXECUTE AS OWNER, SELF, and user.• GRANT and DENY permissions on tables and procedures.• Nesting natively compiled stored procedures• RIGHT OUTER JOIN, LEFT OUTER JOIN, INNER JOIN, and CROSS JOIN in SELECT statements• NOT, OR, and IN operators in SELECT, UPDATE and DELETE statement• UNION ALL and UNION• SELECT DISTINCT• GROUP BY clause with no aggregate functions in the SELECT clause (<select> list).• COLUMNSTORE• COLLATE

Greater Transact-SQL coverage

https://msdn.microsoft.com/en-us/library/ms187926(v=sql.130).aspx





Improved scaling

In-memory OLTP engine has been enhanced to scale linearly on servers up to 4 sockets

Other enhancements include:• Multiple threads to persist memory

optimized tables• Multi-threaded recovery• MERGE operation• Dynamic management view

improvements to sys.dm_db_xtp_checkpoint_stats and sys.dm_db_xtp_checkpoint_files

Data Source=MSSQL; Initial Catalog=AdventureWorks; Integrated Security=SSPI; MultipleActiveResultSets=True

Setup MARS connection for memory optimized tables using the MultipleActiveResultsSets=True in your connection string

Using multiple active result sets (MARS)

In SQL Server 2016 CTP2, the storage for memory-optimized tables will be encrypted as part of enabling TDE on the database

Simply follow the same steps as you would for a disk-based database

Support for Transparent Data Encryption

Windows Operating SystemLevel Data Protection

SQL ServerInstance Level

User DatabaseLevel Database Encryption

Key

Service Master Key

Transparent Database Encryption Architecture

DPAPI encrypts the Service Master Key

MasterDatabase Level Database Encryption

Key

Service Master Key Encrypts the Database master Key for the master Database

Database Master Key of the master Database creates a certificate in the master database

The certificate encrypts the database Encryption Key in the user database

The entire user database is secured by the Datbase Encryption Key (DEK) of the user database by using transparent database encryption

Created at a time of SQL Server setup

Statement:CREAT MASTER KEY…

Statement:CREATE CERTIFICATE…

Statement:CREATE DATABASE ENCRYPTION KEY…

Statement:ALTER DATABSE… SET ENCRYPTION

New Transaction Performance Analysis

Overview report New report replaces the need to use the Management Data Warehouse to analyze which tables and stored procedures are candidates for in-memory optimization

Query StoreYour flight data recorder for your database

Have You Ever…?…had your system down/slowed down and everyone waiting for you to magically fix the problem ASAP?

…upgraded an application to the latest SQL Server version and had an issue with a plan change slowing your application down?

…had a problem with your Azure SQL Database and been unable to determine what was going wrong?

With Query Store…

I CAN get full history of query execution

I CAN quickly pinpoint the most expensive queries

I CAN get all queries that regressed

I CAN easily force better plan from history with a single line of

T-SQL

I CAN safely do server restart or upgrade

Compile MSG

Execute MSG

Query Store

Durability latency controlled by DB option

DATA_FLUSH_INTERNAL_SECONDS

Async Write-Back

Compile

Execute

SQL

Plan Store

Runtime Stats

Query Store

Schema

Query data storeCollects query texts (+ all relevant properties)

Stores all plan choices and performance metrics

Works across restarts / upgrades / recompiles

Dramatically lowers the bar for perf. Troubleshooting

New Views

Intuitive and easy plan forcing

Keeping stability while upgrading to SQL

Sever 2016

Upgrade to SQL vNextKeep

110/120 CompatLev

elFreeze plans

(optional)

Run Query Store

(establish perf.

baseline)

Move to 130

CompatLevel and unfreeze

plans

Monitor perf. and

fix regressions with

plan forcing

Monitoring Performance By Using the

Query Store The query store feature provides DBAs with insight on query plan choice and performance

Working with Query Store

/* (1) Turn ON Query Store */

ALTER DATABASE MyDB SET QUERY_STORE = ON;

/* (2) Review current Query Store parameters */SELECT * FROM sys.database_query_store_options

/* (3) Set new parameter values */ALTER DATABASE MyDBSET QUERY_STORE ( OPERATION_MODE = READ_WRITE, CLEANUP_POLICY = ( STALE_QUERY_THRESHOLD_DAYS = 30 ), DATA_FLUSH_INTERVAL_SECONDS = 3000, MAX_SIZE_MB = 500, INTERVAL_LENGTH_MINUTES = 15);

/* (4) Clear all Query Store data */ALTER DATABASE MyDB SET QUERY_STORE CLEAR;

/* (5) Turn OFF Query Store */ALTER DATABASE MyDB SET QUERY_STORE = OFF;

/* (6) Performance analysis using Query Store views*/SELECT q.query_id, qt.query_text_id, qt.query_sql_text, SUM(rs.count_executions) AS total_execution_countFROMsys.query_store_query_text qt JOIN sys.query_store_query q ON qt.query_text_id = q.query_text_id JOINsys.query_store_plan p ON q.query_id = p.query_id JOINsys.query_store_runtime_stats rs ON p.plan_id = rs.plan_idGROUP BY q.query_id, qt.query_text_id, qt.query_sql_textORDER BY total_execution_count DESC

/* (7) Force plan for a given query */exec sp_query_store_force_plan 12 /*@query_id*/, 14 /*@plan_id*/

DB-level feature exposed through T-SQL extensions

ALTER DATABASE

Catalog views (settings, compile & runtime stats)

Stored Procs (plan forcing, query/plan/stats cleanup)

Live query statisticsCollect actual metrics about query while running

View CPU/memory usage, execution time, query progress, etc.

Enables rapid identification of potential bottlenecks for troubleshooting query performance issues.

Allows drill down to live operator level statistics:

Number of generated rows

Elapsed time

Operator progress

Live warnings, etc.

Java Script Object Notation (JSON)

JSON became ubiquitous

Compact and simple data exchange format

The choice on the web

Recommended scenario

I CAN accept JSON, easily parse and store it as relational

I CAN export relational easily as JSON

I CAN correlate relational and non-relational

Motivation

[ { "Number":"SO43659", "Date":"2011-05-31T00:00:00" "AccountNumber":"AW29825", "Price":59.99, "Quantity":1 }, { "Number":"SO43661", "Date":"2011-06-01T00:00:00“ "AccountNumber":"AW73565“, "Price":24.99, "Quantity":3 }]

Number DateCustomer Price

Quantity

SO43659 2011-05-31T00:00:00 AW29825 59.99 1

SO43661 2011-06-01T00:00:00 AW73565 24.99 3

SELECT * FROM myTable

FOR JSON AUTO

SELECT * FROM OPENJSON(@json)

Data exchange with JSON

CREATE TABLE SalesOrderRecord ( Id int PRIMARY KEY IDENTITY, OrderNumber NVARCHAR(25) NOT NULL, OrderDate DATETIME NOT NULL,

JSalesOrderDetails NVARCHAR(4000) CONSTRAINT SalesOrderDetails_IS_JSON

CHECK ( ISJSON(JSalesOrderDetails)>0 ), Quantity AS CAST(JSON_VALUE(JSalesOrderDetails, '$.Order.Qty') AS int) )GO

CREATE INDEX idxJson ON SalesOrderRecord(Quantity) INCLUDE (Price);

JSON and relational

JSON is plain textISJSON guarantees

consistency

Optimize further with computed column and

INDEX

SELECT t.Id, t.OrderNumber, t.OrderDate,JSON_VALUE(t.JSalesOrderDetails,

'$.Order.ShipDate') AS ShipDateFROM SalesOrderRecord AS tWHERE ISJSON(t.JSalesOrderDetails) > 0AND t.Price > 1000

Plans for CTP3

Ability to load JSON text in tables

Extract values from JSON text

Index properties in JSON text stored in columns

-- and more

Querying JSON (CTP3)

No new data type

If you need to store it raw, store it as NVARCHAR

What is new:

Easy export: FOR JSON

Easy import: OPENJSON

Easy handling: ISJSON, JSON_VALUE

How to handle JSON?

SELECT OrderNumber AS

'Order.Number',OrderDate AS 'Order.Date'

FROM SalesOrder

FOR JSON PATH

For JSON path

[ { "Order":{ "Number":"SO43659", "Date":"2011-05 31T00:00:00“ }, }, { "Order":{ "Number":"SO43660", "Date":"2011-06-01T00:00:00“ }, }]

Query Result (JSON array)

SELECT SalesOrderNumber, OrderDate, UnitPrice, OrderQtyFROM Sales.SalesOrderHeader H INNER JOIN Sales.SalesOrderDetail D ON H.SalesOrderID = D.SalesOrderIDFOR JSON AUTO

For JSON AUTO

[ { "SalesOrderNumber":"SO43659", "OrderDate":"2011-05-31T00:00:00", "D":[ {"UnitPrice":24.99, "OrderQty":1 } ] }, { "SalesOrderNumber":"SO43659" , "D":[ { "UnitPrice":34.40 }, { "UnitPrice":134.24, "OrderQty":5 } ] }]

Query Result

CREATE TABLE OrderRecord ( Id int PRIMARY KEY IDENTITY, OrderDetails NVARCHAR(25)

CONSTRAINT OrderDetails_IS_JSON CHECK (ISJSON(OrderDetails)>1),

Quantity AS CAST(JSON_VALUE(OrderDetails,'$. Order.Qty') AS int))

JSON functions (CTP3)

CREATE INDEX idxJson ON OrderRecord(Quantity);

ISJSON (json_text) JSON_VALUE (json_text,json_path)

OPENJSON

OPENJSON (@json, N'$.Orders.OrdersArray')WITH (

Number varchar(200) N'$.Order.Number',

Date datetime N'$.Order.Date',Customer varchar(200)

N'$.AccountNumber',Quantity int

N'$.Item.Quantity')

{"Orders": { "OrdersArray": [ { "Order": { "Number":"SO43659", "Date":"2011-05-31T00:00:00“ }, "AccountNumber":"AW29825“, "Item": { "Price":2024.9940, "Quantity":1 } }, { "Order":{ “Number":"SO43661", "Date":"2011-06-01T00:00:00“ }, "AccountNumber":"AW73565“, "Item": { "Price":2024.9940, "Quantity":3 } }]} }

Number Date Customer Quantity

SO43659 2011-05-31T00:00:00

AW29825 1

SO43661 2011-06-01T00:00:00

AW73565 3

SQL Server and Azure DocumentDB

Schema-free NoSQL document store

Scalable transactional processing for rapidly

changing apps

Premium relational DB capable to

exchange data with modern apps & services

Derives unified insights from structured/unstructured data

SQL Server 2016

JSON JSJSJSON

Key takeaways

Temporal: data audit and time-based analysis

Query Store: performance troubleshooting and tuning

JSON: interop with modern services and applications

Available in the cloud and on-prem

QUERY STORE

JSON

TemporalQuery back in time

Real data sources are dynamic Historical data may be critical to business success

Traditional databases fail to provide required insights

Workarounds are…Complex, expensive, limited, inflexible, inefficient

SQL Server 2016 makes life easyNo change in programming model

New Insights

Why Temporal

Time Travel Data Audit

Slowly Changing Dimensions

Repair record-level corruptions

No change in programming model

New Insights

INSERT / BULK INSERT

UPDATE

DELETE

MERGE

DML SELECT * FROM temporal

Querying

How to start with temporal

CREATE temporal TABLE PERIOD FOR SYSTEM_TIME…

ALTER regular_table TABLE ADD PERIOD…

DDL

FOR SYSTEM_TIMEAS OF FROM..TOBETWEEN..ANDCONTAINED IN

Temporal Querying

Temporal table (actual data)

Insert / Bulk Insert

* Old versions

Update */ Delete *

How system-time works?

History Table

Temporal table (actual data)

Temporal Queries * (Time travel,etc.)

How system-time works?

History Table

Regular queries (current data)

* Include Historical Version

DepNumDepName

MngrID From To

A001 Marketing 5 2005 2008

A002 Sales 2 2005 2007

A003 Consulting 6 2005 2006

A003 Consulting 10 2009 2012

DepNum DepName

MngrID

A001 Marketing 5

A001 Marketing 6

A002 Sales 2

A002 Sales 5

A003 Consulting 6

A003 Consulting 10

DepNumDepName

MngrID From To

A001 Marketing 6 2008 ∞

A002 Sales 5 2007 ∞A001

A002

A003

period of validity current time

∞

∞

Department (current)

Department (history)

Department (current + history)

2005 2015

SELECT * FROM DepartmentSELECT * FROM Department FOR SYSTEM_TIMEBETWEEN '2006.01.01' AND '2007.01.01'

SELECT * FROM Department FOR SYSTEM_TIMECONTAINED IN ('2007.01.01', '2009.01.01')SELECT * FROM Department FOR SYSTEM_TIME AS OF '2006.01.01'

Getting insights from temporal – AS OF

A001

A002

A003

“Get actual row versions”AS OFBETWEEN..ANDCONTAINED IN

SELECT * FROM Department FOR SYSTEM_TIME AS OF '2010.01.01'

Facts:

1. History is much bigger than actual data

2. Retained between 3 and 10 years

3. “Warm”: up to a few weeks/months

4. “Cold”: rarely queried

Solution:

History as a stretch table:

PeriodEnd < “Now - 6 months”

Azure SQL Database

Provides correct information about stored facts at any point in time, or between 2 points in time.

There are two orthogonal sets of scenarios with regards to temporal data:

System(transaction)-time

Application-time

SELECT * FROM Person.BusinessEntityContactFOR SYSTEM_TIME BETWEEN @Start AND @EndWHERE ContactTypeID = 17

Performance

Temporal database support - BETWEEN









Dynamic data maskingReal-time obfuscation of data to prevent unauthorized access











Always Encrypted

Prevents Data Disclosure

Client-side encryption of sensitive data using keys that are never given to the database system.

Queries on Encrypted Data

Support for equality comparison, incl. join, group by and distinct operators.

Application Transparency

Minimal application changes via server and client library enhancements.

Allows customers to securely store sensitive data outside of their trust boundary.Data remains protected from high-privileged, yet unauthorized users.

Benefits of Always Encrypted

dbo.Patients

Jane Doe

Name

243-24-9812

SSN

USA

Country

Jim Gray 198-33-0987

USA

John Smith 123-82-1095

USA

dbo.Patients

Jane Doe

Name

1x7fg655se2e

SSN

USA

Jim Gray 0x7ff654ae6d

USA

John Smith 0y8fj754ea2c

USA

Country

Result Set

Jim Gray

Name

Jane Doe

Name

1x7fg655se2e

SSN

USA

Country

Jim Gray 0x7ff654ae6d

USA

John Smith 0y8fj754ea2c

USA

dbo.Patients

SQL Server

Query

Trusted

Apps

SELECT Name FROM Patients WHERE SSN=@SSN@SSN='198-33-0987'

Result Set

Jim Gray

Name

SELECT Name FROM Patients WHERE SSN=@SSN

@SSN=0x7ff654ae6d

Column Encryption

Key

EnhancedADO.NET

Library

ColumnMasterKey

Client side

Always EncryptedHelp protect data at rest and in motion, on-premises & cloud

ciphertext

Randomized encryptionEncrypt('123-45-6789') = 0x17cfd50aRepeat: Encrypt('123-45-6789') = 0x9b1fcf32Allows for transparent retrieval of encrypted data but NO operationsMore secure

Deterministic encryptionEncrypt('123-45-6789') = 0x85a55d3fRepeat: Encrypt('123-45-6789') = 0x85a55d3fAllows for transparent retrieval of encrypted data AND equality comparison

E.g. in WHERE clauses and joins, distinct, group by

Two types of encryption available

Randomized encryption uses a method that encrypts data in a less predictable manner

Deterministic encryption uses a method which always generates the same encrypted value for any given plain text value

Types of Encryption for Always Encrypted

Security Officer

1. Generate CEKs and Master Key

2. Encrypt CEK

3. Store Master Key Securely

4. Upload Encrypted CEK to DB

CMK Store:Certificate Store

HSMAzure Key Vault

…

EncryptedCEK

Column Encryption Key(CEK)

ColumnMaster Key(CMK)

Key Provisioning

CMK

DatabaseEncrypted CEK

Param

Encryption Type/ Algorithm

Encrypted CEK Value

CMK Store Provider Name CMK Path

@Name

Non-DET/ AES 256

CERTIFICATE_STORE

Current User/ My/f2260…

EXEC sp_execute_sql N'SELECT * FROM Customers WHERE SSN = @SSN', @params = N'@SSN VARCHAR(11)', @SSN=0x7ff654ae6d

Param

Encryption Type/ Algorithm

Encrypted CEK Value

CMK Store Provider Name CMK Path

@SSN DET/ AES 256

CERTIFICATE_STORE

Current User/ My/f2260…

Enhanced ADO.NET

Plaintext CEKCache

exec sp_describe_parameter_encryption @params = N'@SSN VARCHAR(11)', @tsql = N'SELECT * FROM Customers WHERE SSN = @SSN'

Result set (ciphertext)

Name

Jim Gray

Result set (plaintext)

using (SqlCommand cmd = new SqlCommand("SELECT Name FROM Customers WHERE SSN = @SSN“ , conn)){ cmd.Parameters.Add(new SqlParameter( "@SSN", SqlDbType.VarChar, 11).Value = "111-22-3333"); SqlDataReader reader = cmd.ExecuteReader();

Client - Trusted SQL Server - Untrusted

En

crypti

on

meta

data

Name

0x19ca706fbd9

En

crypti

on

meta

data

CMK Store

Example

Select columns to be encrypted Analyze

schema and application queries to detect conflicts (build time)

Set up the keys: master & CEK

Static schema analysis tool (SSDT only)

UI for selecting columns (no automated data classification)

Key setup tool to automate selecting CMK, generating and encrypting CEK and uploading key metadata to the database

Setup (SSMS or SSDT)User Experience: SSMS or SSDT (Visual Studio)

Existing App – SetupUser Experience: SSMS or SSDT (Visual Studio)

UI for selecting columns (no automated data classification)

Select candidate columns to be encrypted

Analyze schema and application queries to detect conflicts and identify optimal encryption settings

Set up the keys

Encrypt selected columns while migrating the database to a target server (e.g. in Azure SQL Database

Key Setup tool to streamline selecting CMK, generating and encrypting CEK and uploading key metadata to the database

Encryption tool creating new (encrypted) columns, copying data from old (plain text) columns, swapping columns and re-creating dependencies

Select desired encryption settings for selected columns

UI for configuring encryption settings on selected columns (accepting/editing recommendations from the analysis tool)

Schema/workload analysis tool analyzing the schema and profiler logs

Data remains encrypted during query

Summary: Always encryptedProtect data at rest and in motion, on-premises & cloud Capability

ADO.Net client library provides transparent client-side encryption, while SQL Server executes T-SQL queries on encrypted data

Benefits

Sensitive data remains encrypted and queryable at all times on-premises & cloud.

Unauthorized users never have access to data or keys


Apps TCE-enabledADO .NET library

SQL ServerEncrypte

d query

Columnarkey

No app changes

Master key

Row-Level Security.SQL Server 2016SQL Database

Fine-grained access control over specific rows in a database table

Help prevent unauthorized access when multiple users share the same tables, or to implement connection filtering in multitenant applications

Administer via SQL Server Management Studio or SQL Server Data Tools

Enforcement logic inside the database and schema bound to the table.

Protect data privacy by ensuring the right access across rows

Status: Public preview

SQL Database

Customer 1

Customer 2

Customer 3

Row-level security

Fine-grained access control

Keeping multi-tenant databases secure by limiting access by other users who share the same tables.

Application transparency

RLS works transparently at query time, no app changes needed.

Compatible with RLS in other leading products.

Centralized security logic

Enforcement logic resides inside database and is schema-bound to the table it protects providing greater security. Reduced application maintenance and complexity.

Store data intended for many consumers in a single database/table while at the same time restricting row-level read & write access based on users’ execution context.

Benefits of row-level security

CREATE SECURITY POLICY mySecurityPolicyADD FILTER PREDICATE dbo.fn_securitypredicate(wing, startTime,

endTime) ON dbo.patients

Predicate functionUser-defined inline table-valued function (iTVF) implementing security logicCan be arbitrarily complicated, containing joins with other tables

Security predicateApplies a predicate function to a particular table (SEMIJOIN APPLY)Two types: filter predicates and blocking predicates

Security policyCollection of security predicates for managing security across multiple tables

RLS Concepts

Security

CREATE FUNCTION dbo.fn_securitypredicate(@wing int)

RETURNS TABLE WITH SCHEMABINDING AS

return SELECT 1 as [fn_securitypredicate_result] FROM

StaffDuties d INNER JOIN Employees e

ON (d.EmpId = e.EmpId)

WHERE e.UserSID = SUSER_SID()

AND @wing = d.Wing;

CREATE SECURITY POLICY dbo.SecPol

ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing)

ON Patients

WITH (STATE = ON)

Fine-grained access control over rows in a table based on one or more pre-defined filtering criteria, e.g., user’s role or clearance level in organization.

Concepts:Predicate function

Security policy

Example

Two

App user (e.g., nurse) selects from Patients tableThree

Security Policy transparently rewrites query to apply filter predicate

Database Policy Manager

CREATE FUNCTION dbo.fn_securitypredicate(@wing int) RETURNS TABLE WITH SCHEMABINDING AS return SELECT 1 as [fn_securitypredicate_result] FROM StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() AND @wing = d.Wing;

CREATE SECURITY POLICY dbo.SecPol ADD FILTER PREDICATE dbo.fn_securitypredicate(Wing) ON Patients WITH (STATE = ON)

FilterPredicate:

INNER JOIN…

SecurityPolicy

Application

Patients

One

Policy manager creates filter predicate and security policy in T-SQL, binding the predicate to the Patients table

Nurse

SELECT * FROM Patients

SELECT * FROM Patients SEMIJOIN APPLY dbo.fn_securitypredicate(patients.Wing);

SELECT Patients.* FROM Patients, StaffDuties d INNER JOIN Employees e ON (d.EmpId = e.EmpId) WHERE e.UserSID = SUSER_SID() AND Patients.wing = d.Wing;

RLS in Three Steps

-- The following syntax creates a security policy with a filter predicate for the Customer table, and leaves the security policy disabledCREATE SECURITY POLICY [FederatedSecurityPolicy]

ADD FILTER PREDICATE [rls].[fn_securitypredicate]([CustomerId])

ON [dbo].[Customer];

-- Create a new schema and predicate function, which will use the application user ID stored in CONTEXT_INFO to filter rows.CREATE FUNCTION rls.fn_securitypredicate (@AppUserId int)

RETURNS TABLE WITH SCHEMABINDING

AS RETURN (SELECT 1 AS fn_securitypredicate_result WHERE

DATABASE_PRINCIPAL_ID() = DATABASE_PRINCIPAL_ID('dbo') -- application context

AND CONTEXT_INFO() = CONVERT(VARBINARY(128), @AppUserId); GO

Creates a security policy for row level security.

The following examples demonstrate the use of the CREATE SECURITY POLICY syntax.

For an example of a complete security policy scenario, see Row Level Security.

Create Security Policy

https://msdn.microsoft.com/en-us/library/azure/dn765131.aspx

Dynamic Data MaskingSQL Server 2016SQL Database

Configuration made easy in the new Azure portal

Policy-driven at the table and column level, for a defined set of users

Data masking applied in real-time to query results based on policy

Multiple masking functions available (e.g. full, partial) for various sensitive data categories (e.g. Credit Card Numbers, SSN, etc.)

SQL DatabaseSQL Server 2016 CTP2

Table.CreditCardNo

4465-6571-7868-5796

4468-7746-3848-1978

4484-5434-6858-6550

Real-time data masking; partial masking

Status: Public preview

Dynamic Data MaskingPrevent the abuse of sensitive data by hiding it from users

Audit success/failure of database operations


Transparent Data Encryption support for storage of In-memory OLTP Tables

Backup encryption now supported with compression

Other security enhancements

MSDN Documentation https://msdn.microsoft.com/en-us/library/dn765131.aspx

Security Center for SQL Server Database & SQL Database\https://msdn.microsoft.com/en-us/bb510589.aspx

SQL Server Security BlogAdditional examples, useful tips and tricks

http://blogs.msdn.com/b/sqlsecurity/

SQL Server Label Security ToolkitUpdated version to take advantage of RLS coming later in CY15

http://sqlserverlst.codeplex.com/

Security resources

https://msdn.microsoft.com/en-us/library/dn765131.aspx



https://msdn.microsoft.com/en-us/bb510589.aspx

https://msdn.microsoft.com/en-us/bb510589.aspx
















Dynamic Data MaskingReal-time obfuscation of data to prevent unauthorized access











Enhanced AlwaysOn

Greater scalability:Load balancing readable secondaries

Increased number of auto-failover targets

Log transport performance

Improved manageability:DTC support

Database-level health monitoring

Group managed service account

AG_Listener

New York(Primary)

Asynchronous data Movement

Synchronous data Movement

Unified HA Solution

Enhanced AlwaysOn Availability Groups

AG

Hong Kong

(Secondary)

AG

New Jersey

(Secondary)

AG

DR Site Computer2

Computer3

Computer4

Computer5

READ_ONLY_ROUTING_LIST=(('COMPUTER2', 'COMPUTER3', 'COMPUTER4'), 'COMPUTER5')

Primary Site

Computer1 (Primary)

Readable Secondary load balancing

Ability to rebuild online indexes in a single partition provides partition-level control for users who need continual access to the database.

Allows database administrators to specify whether or not to terminate processes that block their ability to lock tables.

SQL Server 2016 CTP2 now provides 100% uptime with enhanced online database operations when conducting ALTER or TRUNCATE operations on tables.

Enhanced online operations






Temporal database supportquery data as points in time



Dynamic Data MaskingReal-time obfuscation of data to prevent unauthorized access











Scalability improvements

Supports caching data with automatic, multiple TempDB files per instance in multi-core environments

Reduces metadata and allocation contention for TempDB workloads, improving performance and scalability

Enhanced database caching

Enhanced support for Windows ServerHardware Acceleration for TDE Encryption/Decryption

Implements Microsoft’s next gen cryptography

Takes advantage of specialized microprocessor instructions

Can improve performance 3x to 10x

Parallelizing the Decryption Built-in Function to Improve Read Performance

Decryption now supported as parallelizable (used to be sequential only)

Results in dramatically better response times for queries with encrypted data columns

Switches

Networking

Compute

Rack 2

Edge

Storage

Networking

Compute

Rack 1

Edge

Storage

Management

Switches

Networking

Compute

Rack 3

Edge

Storage

Switches

Networking

Compute

Rack 4

Edge

Storage

Cloud Platform System advantagesPartnership with Dell.

Combination of optimized hardware/software.

Designed specifically to reduce the complexity and risk of implementing a self-service cloud.

CPS can go from delivery to live within days—not months—and lets service providers and enterprises move up the stack to focus on delivering services to users

Access any data Scale and manage

Powerful Insights

Advanced analytics

PolyBase Insights from data across SQL Server and Hadoop with simplicity of T-SQL

Enhanced SSISDesigner support for previous SSIS versions

Support for Power Query

Enterprise-grade Analysis Services Enhanced performance and scalability for analysis services

Single SSDT in Visual Studio 2015 (CTP3)Build richer analytics solutions as part of your development projects in Visual Studio

Enhanced MDS Excel add-in 15x faster; more granular security roles; archival options for transaction logs; and reuse entities across models

Mobile BIBusiness insights for your on-premises data through rich visualization on mobile devices with native apps for Windows, iOS and Android

Enhanced Reporting ServicesNew modern reports with rich visualizations

R integration (CTP3)Bringing predictive analytic capabilities to your relational database

Analytics libraries (CTP3)Expand your “R” script library with Microsoft Azure Marketplace



Powerful Insights

Advanced analytics





Single SSDT in Visual Studio 2015 Build richer analytics solutions as part of your development projects in Visual Studio







PolyBase for SQL Server 2016

Query relational and non-relational data, on-premises

and in Azure

Apps

T-SQL query

SQL Server

Hadoop

Capability

T-SQL for querying relational and non-relational data across SQL Server and Hadoop

Benefits

New business insights across your data lake

Leverage existing skillsets and BI tools

Faster time to insights and simplified ETL process

PolyBaseQuery relational and non-relational data with T-SQL

Prerequisites for installing PolyBase

64-bit SQL Server Evaluation edition

Microsoft .NET Framework 4.0.

Oracle Java SE RunTime Environment (JRE) version 7.51 or higher

NOTE: Java JRE version 8 does not work.

Minimum memory: 4GB

Minimum hard disk space: 2GB

Using the installation wizard for PolyBase

Run SQL Server Installation Center. (Insert SQL Server installation media and double-click Setup.exe.)

Click Installation, then click New Standalone SQL Server installation or add features

On the feature selection page, select PolyBase Query Service for External Data.

On the Server Configuration Page, configure the PolyBase Engine Service and PolyBase Data Movement Service to run under the same account.

-- Run sp_configure ‘hadoop connectivity’ -- and set an appropriate valuesp_configure

@configname = 'hadoop connectivity', @configvalue = 7;

GORECONFIGUREGO

-- List the configuration settings for -- one configuration namesp_configure @configname='hadoop connectivity';GO

Option values

0: Disable Hadoop connectivity

1: Hortonworks HDP 1.3 on Windows Server Azure blob storage (WASB[S])

2: Hortonworks HDP 1.3 on Linux

3: Cloudera CDH 4.3 on Linux

4: Hortonworks HDP 2.0 on Windows Server Azure blob storage (WASB[S])

5: Hortonworks HDP 2.0 on Linux

6: Cloudera 5.1 on Linux

7: Hortonworks 2.1 and 2.2 on Linux Hortonworks 2.2 on Windows Server Azure blob storage (WASB[S])

Choose Hadoop data source with sp_configure

Start the PolyBase services

After running for sp_configure, you must stop and restart the SQL Server engine service

Run services.msc

Find the services shown below and stop each one

Restart the services

-- Using credentials on database requires enabling-- traceflag DBCC TRACEON(4631,-1)

-- Create a master keyCREATE MASTER KEY ENCRYPTION BY PASSWORD = 'S0me!nfo';

CREATE CREDENTIAL WASBSecret ON DATABASE WITH IDENTITY = 'pdw_user', Secret = 'mykey==';

-- Create an external data source (Azure Blob Storage)-- with the credentialCREATE EXTERNAL DATA SOURCE Azure_Storage WITH ( TYPE = HADOOP, LOCATION ='wasb[s]://[email protected]/path’,

CREDENTIAL = WASBSecret)

Type methods for providing credentials

Core-site.xml in installation path of SQL Server - <SqlBinRoot>\Polybase\Hadoop\Conf

Credential object in SQL Server for higher security

NOTE: The syntax for a database-scoped credential (CREATE CREDENTIAL … ON DATABASE) is temporary and will change in the next release. This new feature is documented only in the examples in the CTP2 content, and will be fully documented in the next release.

Configure PolyBase for Azure blob storage

-- Create an external data source (Hadoop)CREATE EXTERNAL DATA SOURCE hdp2 with (TYPE = HADOOP, LOCATION ='hdfs://10.xxx.xx.xxx:xxxx', RESOURCE_MANAGER_LOCATION='10.xxx.xx.xxx:xxxx')

CTP2 supports the following Hadoop distributions

Hortonworks HDP 1.3, 2.0, 2.1, 2.2 for both Windows and Linux

Cloudera CDH 4.3, 5.1 on Linux

Create a reference to a Hadoop cluster

-- Create an external file format -- (delimited text file)CREATE EXTERNAL FILE FORMAT ff2 WITH (FORMAT_TYPE = DELIMITEDTEXT, FORMAT_OPTIONS (FIELD_TERMINATOR ='|', USE_TYPE_DEFAULT = TRUE))

CTP2 supports the following file formats

Delimited text

Hive RCFile

Hive ORC

Define the external file format

-- Create an external table pointing to file stored in HadoopCREATE EXTERNAL TABLE [dbo].[CarSensor_Data] ( [SensorKey] int NOT NULL, [CustomerKey] int NOT NULL, [GeographyKey] int NULL, [Speed] float NOT NULL, [YearMeasured] int NOT NULL)WITH (LOCATION='/Demo/car_sensordata.tbl', DATA_SOURCE = hdp2, FILE_FORMAT = ff2, REJECT_TYPE = VALUE, REJECT_VALUE = 0

The external table provides a T-SQL reference to the data source used to:

Query Hadoop or Azure blob storage data with Transact-SQL statements

Import and store data from Hadoop or Azure blob storage into your SQL Server database

Create an external table to the data source

-- Create statistics on an external table. CREATE STATISTICS StatsForSensors ON CarSensor_Data(CustomerKey, Speed)

In CTP2, you can optimize query execution against the external tables using statistics

Optimize queries by adding statistics

SELECT DISTINCT C.FirstName, C.LastName, C.MaritalStatusFROM Insurance_Customer_SQL INNER JOIN (

SELECT * FROM SensorData_ExternalHDP WHERE Speed > 35UNION ALLSELECT * FROM SensorData_ExternalHDP2 WHERE Speed > 35) AS SensorDON C.CustomerKey = SensorD.CustomerKey

External tables referring to data in 2 HDP Hadoop clusters

SQL Server table

Query Capabilities (1)Joining relational and external data

SELECT DISTINCT C.FirstName, C.LastName, C.MaritalStatusFROM Insurance_Customer_SQL -- table in SQL Server…OPTION (FORCE EXTERNALPUSHDOWN) – push-down computationCREATE EXTERNAL DATA SOURCES ds-hdp WITH .( TYPE = Hadoop, LOCATION = “hdfs://10.193.27.52:8020”, Resources_Manager_Location = ‘10.193.27.52:8032’);

Pushing ComputeEither on data source level or Per-query basis using new query hints

Query Capabilities (2)Push-Down Computation

SSIS improvements

SSIS improvements for SQL Server 2016

CTP2

AlwaysOn support

Incremental deployment of packages

Improved project upgrade support

CTP3

Designer improvements

One designer multi-version support

OData V4 support

Power Query as a data source

AlwaysOn Availability

Groups Secondary for

SSISDBNew York(Primary)

New Jersey

(Secondary)

SSIS DB

SSISDB

SQL Server 2012SSIS Project XSQL Server 2016SSIS Project X

Improved project upgrade

AlwaysOn for SSIS Catalog

Steps to configure AlwaysOn for SSIS catalog

Create Integrations Services catalog

Add SSISDB to an AlwaysOn Availability Group

Enable SSIS support for AlwaysOn

private static void Main(string[] args) { // Connection string to SSISDB var connectionString = "Data Source=.;Initial Catalog=SSISDB;Integrated Security=True;MultipleActiveResultSets=false";

using (var sqlConnection = new SqlConnection(connectionString)) { sqlConnection.Open();

var sqlCommand = new SqlCommand { Connection = sqlConnection, CommandType = CommandType.StoredProcedure, CommandText = "[catalog].[deploy_packages]" };

var packageData = Encoding.UTF8.GetBytes(File.ReadAllText(@"C:\Test\Package.dtsx"));

// DataTable: name is the package name without extension and package_data is byte array of package. var packageTable = new DataTable(); packageTable.Columns.Add("name", typeof(string)); packageTable.Columns.Add("package_data", typeof(byte[])); packageTable.Rows.Add("Package", packageData);

// Set the destination project and folder which is named Folder and Project. sqlCommand.Parameters.Add(new SqlParameter("@folder_name", SqlDbType.NVarChar, ParameterDirection.Input, "Folder", -1)); sqlCommand.Parameters.Add(new SqlParameter("@project_name", SqlDbType.NVarChar, ParameterDirection.Input, "Project", -1)); sqlCommand.Parameters.Add(new SqlParameter("@packages_table", SqlDbType.Structured, ParameterDirection.Input, packageTable, -1));

var result = sqlCommand.Parameters.Add("RetVal", SqlDbType.Int); result.Direction = ParameterDirection.ReturnValue;

sqlCommand.ExecuteNonQuery(); } }

Deployment options in CTP2

Integration Services Deployment Wizard

SQL Server Management Studio

deploy_packages stored procedure

Object model API

Deployment options in CTP3

SQL Server Data Tools for Business Intelligence

Deploy packages to Integration Services server


Powerful Insights

Advanced analytics





Single SSDT in Visual Studio 2015 (CTP3)Build richer analytics solutions as part of your development projects in Visual Studio







Enterprise grade Analysis Services

Scale your tabular models with support for high end servers

More memory

More cores

Enhanced Analysis ServicesDeliver high performance and scalability for your BI solutions

High-end server hardware

Capability

Parallel partition processing

NUMA optimization for tabular models

On-demand loading and paging

Tabular and MOLAP modeling enhancements

Benefits

Better performance and higher scale

Lower TCO – more efficient utilization of existing hardware capabilities


SSAS Enterprise Readiness: MOLAP

Netezza as a data source

Performance improvements (unnatural hierarchies, ROLAP distinct count, drill-through, semi-additive measures)

Detect MOLAP index corruption using DBCC

High-end Server Hardware

SSAS Enterprise Readiness: Tabular

Processing optimizations

Parallel partition processing

Support for Power Query in AS engine

Support for BI directional cross filtering (M:M)

Calculated tables, Date Table

Translations

Tabular scripting language

New DAX functions (DATEDIFF, GEOMEAN, PERCENTILE, PRODUCT, XIRR, XNPV, many more..)

Query engine optimizations

New DAX functions for high perf. reporting

Direct Query perf. optimizations

DimDepartmentGr…DimDepartmentGroupKey…ParentDepartmentGr…

DepartmentGroupNa…

DimAccount

AccountKey

ParenAccountKey

AccountCodeAlter…

ParenAccountCod…

AccountDescription

FactFinance

FinanceKey

DateKey

OrganizationKey

DepartmentGroup…

ScenarioKey

DimOrganization

OrganizationKey

ParnentOrganization

PercentageOfOwn…

OrganizationName

CurrencyKey

DimAccount

ScenarioKey

ScenarioName

*

*

1

*

1

*1

*1

SSAS Rich Modeling Platform: Tabular

Unified Visual Studio Data Tools (CTP3)

Capability

SSDT-BI and SSMS for Visual Studio 2015

Rich data modeling enhancements

New DAX analytical functions

MDX Query Plan tool for performance optimizations and troubleshooting

Benefits

Lower BI solution development costs and faster delivery cycles

Support for most recent version of Visual Studio to create BI solutions

Lower DAX expression complexity through new DAX and SuperDAX functions

{JSON}

Powerful Insights

Rich BI Application PlatformDevelop and deliver BI solutions faster and at lower costs

Master Data Services improvements

Performance and Scale

Overall performance and scale improvements

Target of 15x performance improvement for Excel add-in

Increased performance for bulk entity based staging operations

Security Improvements

New security roles for more granular permissions around read, write and delete

Multiple system administrators

Manageability and ModelingArchival of Transaction Logs

Reuse of entities across models

Support for compound keys

Allow the Name and Code attributes to be renamed

MDS Improvements


Powerful Insights

Advanced analytics












Mobile BI improvements

Mobile BI apps for SQL Server – formally

Datazen For on-premises implementations - optimized for SQL Server.

Rich, interactive data visualization on all major mobile platforms

No additional cost for SQL Server Enterprise Edition customers 2008 or later and Software Assurance

Data visualization and publishing

Create beautiful visualizationsand KPIs with a touch-based designer

Connect to the Mobile BI for SQL Server server toaccess SQL Server data

Publish for access by others

Datazen architecture overviewServer, Publisher and Mobile apps

SQL Server

Analysis Services

File

Data sources

Active directory controller

Authentication

Internet boundary

Viewer

Apps

Publisher

App

Web

browser

Datazen Enterprise Server

Core Service

Data Acquisition Service

Web applications

Enterprise environment

https://itunes.apple.com/us/app/datazen/id604097486

https://play.google.com/store/apps/details?id=com.componentart.datazenclient

http://apps.microsoft.com/windows/en-us/app/datazen-publisher/a23f9233-bcda-49e3-bab4-5e8366b96a37

http://www.windowsphone.com/en-us/store/app/datazen/d887bbbe-e249-4ada-bd22-3f11e2120d9e

http://apps.microsoft.com/windows/en-us/app/datazen-publisher/a23f9233-bcda-49e3-bab4-5e8366b96a37

Mobile BI for SQL Server summary

Beautiful visualizations

KPI repository

Create once - publish to any device

Native apps for all platforms

Perfect scaling to any form factor

Custom branding

Collaborate on the go

Enhanced Reporting Services

Modern reports with SQL Server Reporting

Services Report consumption from modern browsers

Improved parameters

Modern themes

New chart types


Powerful Insights

Advanced analytics











Deeper insights across data (CTP3)

R integration with database engine (CTP3)

Example SolutionsFraud detection

Sales forecasting

Warehouse efficiency

Predictive maintenance

Extensibility

Microsoft AzureMachine Learning Marketplace

New R scripts

010010

100100

010101

010010

100100

010101

010010

100100

010101

010010

100100

010101

Built-in advanced analytics (CTP3)In-database analytics

Built-in to SQL Server

Analytic Library

T-SQL Interface

Relational Data

010010

100100

010101010010

100100

010101

Data ScientistInteract directlywith data

Data Developer/DBAManage data and analytics together

?R

R Integration

Capability

Extensible In-database analytics, integrated with R, exposed through T-SQL

Centralize enterprise library for analytic models

Benefits

No data movement, resulting in faster time

to insights

Real-time analytics on transactional data

Integrate with existing workflows

Unified governance across analytics & storage

SQL Server

Analytical EnginesFull R IntegrationFully Extensible

Data Management Layer

Relational Data

T-SQL InterfaceStream Data In-Memory

Analytics LibraryShare and CollaborateManage and Deploy

R +

Data Scientists

Business Analysts

Publish algorithms, and interact directly with Data

Analysis through TSQL, tools, and vetted

algorithms

DBAsManage storage and analytics together

Advanced Analytics (CTP3)Faster analytics with In-database processing

Hybrid solutions Simplicity Consistency

Stretch DatabaseStretch operational tables in a secure manner into Azure for cost effective historic data availability works with Always Encrypted and Row Level Security

Power BI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall

Hybrid Scenarios with SSISAzure Data Factory integration with SSIS, package lineage and impact analysis and connect SSIS to cloud data source

Enhanced Backup to AzureFaster restore times and 50% reduction in storage, support larger DBs with Block blobs and custom backup schedule with local staging

Easy migration of on-premises SQL ServerSimple point and click migration to Azure

Simplified Add Azure Replica Wizard Automatic listener configuration for AlwaysOn in Azure VMs

Common development, management and identity toolsIncluding Active Directory, Visual Studio, Hyper-V and System Center

Consistent Experience from SQL Server on-premises to Microsoft Azure IaaS and PaaS


Stretch Database

Capability

Stretch large operational tables from on-premises to Azure with the ability to query

Benefits

Cost effective historical data availability for operational DBs

Entire table is online and queryable

Transparent to application

Supports new TCE transparent encryption and Row Level Security

Apps continue working without change or disruption

BI integration for on-

premises and cloud

Cold/closed data

OrdersIn-memory OLTP table

Hot/active data

Order historyStretched table

Trickle data movement and

remote query processing

On-premises

Azure

Stretch SQL Server into AzureSecurely stretch cold tables to Azure with remote query processing

Stretch database architecture

How it works

Creates a secure linked server definition in the on-premises SQL Server

Linked server definition has the remote endpoint as the target

Provisions remote resources and begins to migrate eligible data, if migration is enabled

Queries against tables run against both the local database and the remote endpoint

Remote Endpoint

Remote Data

On-Premise Instance

Azure

Inte

rnet B

oundary

Local Database

Local Data

Eligible Data

-- Enable local serverEXEC sp_configure 'remote data archive' , '1';RECONFIGURE;

-- Provide administrator credential to connect to-- Azure SQL DatabaseCREATE CREDENTIAL <server_address> WITH

IDENTITY = <administrator_user_name>, SECRET = <administrator_password>

-- Alter database for remote data archiveALTER DATABASE <database name>

SET REMOTE_DATA_ARCHIVE = ON (SERVER = server name); GO

-- Alter table for remote data archiveALTER TABLE <table name> ENABLE REMOTE_DATA_ARCHIVE

WITH ( MIGRATION_STATE = ON );GO;

High level steps

Configure local server for remote data archive

Create a credential with administrator permission

Alter specific database for remote data archive

Alter table for remote data archive

Typical workflow to enable Stretch Database

On-premises integration with Power BI

Live dashboards

and exploration

Analysis Services on-

premisesTabular model

Interactive query

AS Connector

Capability

Publish on-premises Analysis Services data models for consumption in Power BI

Benefits

Faster time to solution without having to move your data to the cloud.

Optimized query performance for sub-second response times.

Manage and secure data residing on-premises in SQL Server Analysis Services.

SQL Server vNext

Connect live to on-premises Analysis Services data

SSIS for Azure

SSIS Improvements for Azure services (CTP3)

Capability

Stretch large operational tables from on-premises to Azure with the ability to query

Benefits

Cost effective historical data availability for operational DBs

Entire table is online and queryable

Transparent to application

Supports new TCE transparent encryption and Row Level Security

SQL

Enhanced backup

Managed backupGranular control of the backup schedule;

Local staging support for faster recovery and resilient to transient network issues;

Support for system databases;

Support simple recovery mode.

Backup to Azure block blobsCost savings on storage;

Significantly improved restore performance; and

More granular control over Azure storage.

Azure Storage snapshot backupFastest method for creating backups and running restores

Uses SQL Server db files on Azure Blob Storage

Enhanced backup to Azure



PowerBI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall








Migrate databases to Azure

User DB System Objects SQL Settings

Schema

Data

Logins

Jobs

Certificates

…

Max server memory

Max worker threads

Default language

…

Traceflags

Migration Wizard

On Premise

EZ Button – Migrate to AzureSimple Single Click Migration Experience

Capability

Publish on-premises Analysis Services data models for consumption in Power BI

Benefits

Faster time to solution without having to move your data to the cloud.

Optimized query performance for sub-second response times.

Manage and secure data residing on-premises in SQL Server Analysis Services.

Method 1 Method 2 Method 3

TargetDB

Azure SQL Database

SQL Server

SSMS

2. Import

1. DeployOR

.bacpac

1. Export

Source DB

TargetDB

Azure SQL Database

SQL Server

SQL Azure Migration Wizard

2. Execute

1. Generate

T-SQL

Source DB

TargetDB

Azure SQL Database

SQL Server

SSMS

6. Export/Importor Deploy

SQLAzureMigrationWizard

4. Copy

CopyDB

Source DB

5. Publish

1. ImportVisual Studio

Database Project

2. Transform

3. Edit, Build & Test

*.sql

4. Publish (schema only)

Migrate a compatible database using SSMS

Migrate a near-compatible database using SAMW

Update the database schema offline using Visual Studio and SAMW, and then deploy it with SSMS

Migration methodologies

Migration Cookbook

Migrate an on-premises SQL Server database to Azure SQL Database (v12).

The Migration Cookbook describes various approaches you can use to migrate an on-premises SQL Server database to the latest Azure SQL Database Update (v12).

Download: http://aka.ms/azuresqlmigration

Simplified AlwaysOn with replicas on Azure

Today this requires manually configuring the Listener

SQL Server 2016 CTP2

Allows configuring the Availability Group Listener in Azure

Clients can connect to the Azure replica after failover using the Listener name

Simplified Add Azure Replica WizardAutomatic Listener Configuration



PowerBI with on-premises dataNew interactive query with Analysis Services. Customer data stays behind your firewall








Consistent tools and experience

Consistent ToolsConsistency across:

On-premises, Private Cloud, Public Cloud

SQL Server local, VM, Azure SQL Database

Scalability, availability, security, identity, backup and restore, and replication

Plethora of data sources

Reporting, integration, processing, and analytics

All of this support Hybrid Cloud

Options:

SQL Server on physical machines

SQL Server in on-premises VM (private cloud)

SQL Server in Azure VM (public cloud)

Azure SQL Database (public cloud)

SQL Server, Azure VMs, Azure SQL DB

Shared Lower Cost

Dedicated Higher Cost

Higher Administration Lower Administration

Off Premises

Hybrid Cloud

Physical

SQL ServerPhysical Machines(raw Iron)SQ

L

Virtual

SQL Server Private CloudVirtualized Machines + Appliances

Infrastructure as a service

SQL Server in Azure VMVirtualized machinesSQ

L

Platform as a service

Software as a services

Azure SQL DatabaseVirtualized Databases

SQL

On Premises

The Microsoft data platform

Internal & external

DashboardsReports Ask Mobile

Information managementOrchestration

Extract, transform, load Prediction

RelationalNon-

relational Analytical

Apps

Streaming

VISUALIZE + DECIDE

TRANSFORM + ANALYZE

COLLECT + MANAGE

© 2015 Microsoft Corporation. All rights reserved. Microsoft, Windows, and other product names are or may be registered trademarks and/or trademarks in the U.S. and/or other countries. The information herein is for informational purposes only and represents the current view of Microsoft Corporation as of the date of this presentation. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information provided after the date of this presentation. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS PRESENTATION.

sql server 2016 ctp technical deep dive

Technology