Joydip Kanjilal Microsoft MVP (2007 till 2012), Author and Speaker

Principal Architect (SenecaGlobal IT Services Private Limited)

New features in SQL Server 2016

Agenda

SQL Server 2016 Enhancements Improved Support for OLTP JSON Support Row Level Security Always Encrypted Polybase Stretch Database Query Store

SQL Server 2014 Enhancements In-memory OLTP Engine Transactional performance gains Support for memory-Optimized Tables Support for Natively Compiled SPs Buffer Pool Extensions Clustered ColumnStore Indexes Delayed Transactions Support for Cardinality Estimation

What is SQL Server 2016?

SQL Server 2016 delivers breakthrough mission-critical capabilities with in-memory performance and operational analytics built-in. It delivers a complete database platform for hybrid cloud, enabling you to easily build, deploy and manage solutions that span on-premises and cloud. Comprehensive security features like new Always Encrypted technology help protect your data at rest and in motion, and a world-class high availability and disaster recovery solution adds new enhancements to AlwaysOn technology.

What is SQL Server 2016?

Organizations will gain deeper insights into all of their data with new capabilities that go beyond business intelligence to perform advanced analytics directly within their database and present rich visualizations for business insights on any device. You can use SQL Server 2016 to: Scale up to 12 TB of memory and 640 logical processors with Windows Server

2016 Reach up to 30x faster transactions and 100x faster queries with enhanced in-

memory performance Run real-time Operational Analytics over transactional data Balance loads across readable secondaries in Always On availability groups

Why SQL Server 2016?

Enhanced in-memory performance provides up to 30x faster transactions, more than 100x faster queries than disk-based relational databases and real-time operational analytics

New Always Encrypted technology helps protect your data at rest and in motion, on-premises and in the cloud, with master keys sitting with the application, without application changes

Stretch Database technology keeps more of your customer’s historical data at your fingertips by transparently stretching your warm and cold OLTP data to Microsoft Azure in a secure manner without application changes

Why SQL Server 2016?

Built-in advanced analytics provide the scalability and performance benefits of building and running your advanced analytics algorithms directly in the core SQL Server transactional database.

Business insights through rich visualizations on mobile devices with native apps for Windows, iOS and Android.

Simplify management of relational and non-relational data by querying both with T-SQL using PolyBase.

Faster hybrid backups, high availability and disaster recovery scenarios to back up and restore your on-premises databases to Microsoft Azure and place your SQL Server AlwaysOn secondaries in Azure.

What’s New in SQL Server 2016?

SQL Server 2016 has many new features. Some of features are enhancements to existing features, while others are entirely new features.

JSON Support Row Level Security Always Encrypted Polybase Stretch Database Query Store

JSON Support

JSON is a popular textual data format used for exchanging data in modern web and mobile applications. JSON is also used for storing unstructured data in log files or NoSQL databases like Microsoft Azure DocumentDB. Many REST web services return results formatted as JSON text or accept data formatted as JSON. Most of the Azure services such as Azure Search, Azure Storage, Azure DocumentDb have REST endpoints that return or consume JSON. JSON is the main format for exchanging data between web pages and web servers using AJAX calls.

JSON Support

JSON stands for Java Script Object Notation. With SQL Server 2016 you can now interchange JSON data between applications and the SQL Server database engine. By adding this support Microsoft has provided SQL Server the ability to parse JSON formatted data so it can be stored in a relation format. Additionally, with JSON support you can take relational data, and turn it into JSON formatted data. Microsoft has also added some new functions to provided support for querying JSON data stored in SQL Server. Having these additional JSON features built into SQL Server should make it easier for applications to exchange JSON data with SQL Server.

Built-in functions for JSON processing

ISJSON (jsonText) that checks is the NVARCHAR text input properly formatted according to the JSON specification. You can use this function the create check constraints on NVARCHAR columns that contain JSON text

JSON_VALUE ( jsonText, path) that parses jsonText and extracts scalar value on the specified JavaScript-like path

FORJSON (jsonText) is used to import JSON data into SQL Server, or to convert JSON data into rows and columns.

ISJSON

The ISJSON function lets you test whether a text string is correctly formatted JSON. The ISJSON function returns 1 if a string is valid JSON, otherwise returns 0. The only exception to this is if the string is null, in which case the function returns null. SELECT CASE WHEN ISJSON(@json) > 0 THEN 'The variable value is JSON.' ELSE 'The variable value is not JSON.' END;

FORJSON

One of the JSON-related features supported in SQL Server 2016 is the ability to return data in the JSON format, which we do by adding the FOR JSON clause to a SELECT statement. USE AdventureWorks Go SELECT FirstName, MiddleName, LastName, EmailAddress, PhoneNumber FROM HumanResources.vEmployee WHERE BusinessEntityID in (2, 3) FOR JSON AUTO

OPENJSON

To return a JSON snippet as rowset data, we use the OPENJSON rowset function to convert the data to a relational format. The function returns three values:- 1. key: Property name within the object or index of the element within the

array. 2. value: Property value within the object or value of the array element

specified by the index. 3. type: Value's data type, represented numerically as shown in the

diagram.

Numeric value

Data type

0 null

1 string

2 int

3 true or false

4 array

5 object

OPENJSON

DECLARE @json NVARCHAR(MAX) = N' { "FirstName":null, "LastName":"Duffy", "NatID":245797967, "Current":false, "Skills":["Dev","QA","PM"], "Region":{"Country":"Canada","Territory":"North America"} }'; SELECT * FROM OPENJSON(@json);

Row Level Security

With Row Level Security the SQL database engine will be able to restrict access to row data, based on a SQL Server login. Restricting rows will be done by filter predicates defined in inline table value function. Security policies will ensure the filter predicates get executed for every SELECT or DELETE operation. Implementing row level security at the database layer means application developers will no longer need to maintain code to restrict data from some logins, while allowing other logins to access all the data. With this new feature, when someone queries a tables that contains row level security they will not even know whether or not any rows of data were filtered out.

Row Level Security Concepts

There are three core concepts used in Row-Level Security: Predicate function - An inline table-valued function that implements access

control logic (for example, returning a row or not depending on the principal_name, role, or other attributes of the calling user).

Security predicate - Glues a predicate function to a table (for example, applying a function that checks for rep name or the rep manager role to the Accounts table).

Security policy - A set of security predicates (for example, you might have an Account policy that applies multiple security predicates to Account-related tables, and an HR policy that applies several security predicates to various HR tables).

PolyBase

PolyBase allows you to query distributed data sets. With the introduction of PolyBase you will be able to use Transact SQL statements to query Hadoop, or SQL Azure blob storage. By using PolyBase you can now write adhoc queries to join relational data from SQL Server with semi-structured data stored in Hadoop, or SQL Azure blob storage. This allows you to get data from Hadoop without knowing the internals of Hadoop. Additionally you can leverage SQL Server’s on the fly column store indexing to optimize your queries against semi-structured data. As organizations spread data across many distributed locations, PolyBase will be a solution for them to leverage SQL Server technology to access their distributed semi-structured data.

Stretch Database

The Stretch Database feature provides you a method to stretch the storage of your On-Premise database to Azure SQL Database. But having the stretch database feature allows you to have your most frequently accessed data stored On-Premise, while your less accessed data is off-site in an Azure SQL databases. When you enable a database to “stretch” the older data starts moving over to the Azure SQL database behind the scenes. When you need to run a query that might access active and historical information in a “stretched” database the database engine seamlessly queries both the On-Premise database as well as Azure SQL database and returns the results to you as if they had come from a single source.

Query Store

The Query Store is a feature that can help us to troubleshoot query performance, as well as capturing query execution information, such as the number of executions or the average duration of a query. The Query Store also captures every Execution Plan that has been generated for a specific query and you can then decide what Execution Plan you want SQL Server to use for that specific query. ALTER DATABASE SSWUGDB SET QUERY_STORE = ON GO

Why Should I use Query Store?

Quickly find and fix a plan performance regression by forcing the previous query plan. Fix queries that have recently regressed in performance due to execution plan changes.

Determine the number of times a query was executed in a given time window, assisting a DBA in troubleshooting performance resource problems.

Identify top n queries (by execution time, memory consumption, etc.) in the past x hours.

Audit the history of query plans for a given query. Analyse the resource (CPU, I/O, and Memory) usage

patterns for a particular database.

Always Encrypted

That encryption key is never passed to SQL Server. By doing this you can keep those nosey Database or Windows Administrators from poking around sensitive client application data In-Flight or At-Rest. This feature will now allow you to sleep at night knowing your confidential data stored in a cloud managed database is always encrypted and out of the eyes of your cloud provider.

Multiple TempDB Database Files

It has been a best practice for a while to have more than one tempdb data file if you are running on a multi-core machine. In the past, up through SQL Server 2014, you always had to manually add the additional tempdb data files after you installed SQL Server. With SQL Server 2016 you can now configure the number of tempdb files you need while you are installing SQL Server. Having this new feature means you will no longer need to manually add additional tempdb files after installing SQL Server.

Temporal Table

A temporal table is table that holds old versions of rows within a base table. By having temporal tables SQL Server can automatically manage moving old row versions to the temporal table every time a row in the base table is updated. The temporal table is physically a different table then the base table, but is linked to the base table. If you’ve been building or plan to build your own method to managing row versioning then you might want to check out the new temporal tables support in SQL server 2016 before you go forth and build your own row versioning solution.

In-Memory OLTP

In-Memory OLTP, also known as "Hekaton" and "In-Memory Optimization", is Microsoft’s latest in-memory processing technology. In-Memory OLTP is optimized for Online Transaction Processing (OLTP). It is integrated into SQL Server’s Database Engine and can be used in the exact same manner as any other Database Engine component. In-Memory OLTP originally shipped with SQL Server 2014 and it mainly features two new data structures which are Memory-Optimized Tables, and Natively-Compiled Stored Procedures.

In-Memory OLTP Engine

SQL Server 2016 includes the new in-memory OLTP engine - an in memory engine that facilitates faster application performance. If your application is facing serious concurrency issues with thousands of simultaneous connections trying to lock data, Hekaton offers an intriguing solution. This feature was first introduced in SQL Server 2014 and is arguably getting the most attention from SQL Server watchers, as it promises to drastically speed up SQL Server response times for frequently-accessed data, also known in database parlance as "hot data."

Memory Optimized Tables

Memory-optimized tables store their data into memory using multiple versions of each row’s data. This technique is characterized as ‘non-blocking multi-version optimistic concurrency control’ and eliminates both locks and latches, thereby achieving significant performance advantages. The main features of memory-optimized tables include: Rows in the table are read from, and written to, memory The entire table resides in memory Non-blocking multi-version optimistic concurrency control The option of durable & non-durable data A second copy is maintained on disk for durability (if enabled) Data in memory-optimized tables is only read from disk during database

recovery It is interoperable with disk-based tables

Creating Memory Optimized Tables

--Disk-Based Table CREATE TABLE [dbo].[Product] ( ID INT NOT NULL PRIMARY KEY, Code VARCHAR(10) NOT NULL , Description VARCHAR(200) NOT NULL , Price FLOAT NOT NULL ); GO

Creating Memory Optimized Tables

--Memory-Optimized Non-Durable Table CREATE TABLE [dbo].[Product] ( ID INT NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 1000), Code VARCHAR(10) COLLATE Latinl_General_100_BIN2 NOT NULL, Description VARCHAR(200) NOT NULL, Price FLOAT NOT NULL )WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_ONLY); GO

Creating Memory Optimized Tables

--Memory-Optimized Durable Table CREATE TABLE [dbo].[Product] ( ID INT NOT NULL PRIMARY KEY NONCLUSTERED HASH WITH (BUCKET_COUNT = 1000), Code VARCHAR(10) NOT NULL, Description VARCHAR(200) NOT NULL, Price FLOAT NOT NULL )WITH (MEMORY_OPTIMIZED = ON, DURABILITY = SCHEMA_AND_DATA); GO

Natively Compiled Stored Procedure

A natively-compiled stored procedure is a SQL Server object that can access only memory-optimized data structures such as memory-optimized tables, table variables, etc. Natively compiled stored procedures allow for efficient execution of the queries and business logic in the stored procedure. The main features of a natively-compiled stored procedure include: It is compiled to native code (DLL) upon its creation (the interpreted stored

procedures are compiled at first execution) Aggressive optimizations take time at compile time It can only interact with memory-optimized tables The call to a natively-compiled stored procedure is actually a call to its DLL

entry point

Natively Compiled Stored Procedure

Natively Compiled Stored Procedures are T-SQL procedures compiled to native code and access memory optimized tables. They allow for the efficient execution of business logic and T-SQL queries within the procedure. Stored Procedures marked as NATIVE_COMPILATION are Natively Compiled Stored Procedures. Natively Compiled Stored Procedures are compiled when they are created. On the contrary, Disk-based Stored Procedures are interpreted and are compiled the first time they are executed.

Natively Compiled Stored Procedure

CREATE PROCEDURE [dbo].[spProductUpdate] WITH NATIVE_CCMPILATION, SCHEMABINDING, EXECUTE AS OWNER AS BEGIN ATOMIC WITH ( TRANSACTION ISOLATION LEVEL = SNAPSHOT, LANGUAGE = N’us_english’ ) UPDATE dbo.Product SET Price = Price - ( Price * 0.10 ); END;

Always Encrypted

With the Always Encrypted feature enabled your SQL Server data will always be encrypted within SQL Server. Access to encrypted data will only be available to the applications calling SQL Server. Always Encrypted enables client application owners to control who gets access to see their applications confidential data. It does this by allowing the client application to be the one that has the encryption key.

Enhancements in SQL Server 2014

What’s New in SQL Server 2014?

SQL Server 2014 is the next generation of relational database management system from Microsoft that helps you build high performing, scalable next generation applications that have the capability of leveraging the hybrid cloud platform. You can use SQL Server 2014 to protect, unlock, and scale your mission critical data. SQL Server 2014 provides much improved performance – this is achieved mainly through the support for memory optimized database tables and natively compiled stored procedures.

What’s New in SQL Server 2014?

Microsoft SQL Server 2014 has some great new features that allows you to develop higher performing, more scalable next-generation applications using the hybrid cloud. SQL Server 2014 promises to bring immense performance gains to online transactional processing systems. Microsoft's objective of SQL Server 2014 include: Mission-critical performance Faster insights from any data Platform for Hybrid Cloud

Support for Memory-Optimized Tables

The new Memory-Optimized table feature of SQL Server 2014 allows us to safely create tables in memory for improved performance. A second copy of the table exists on disk and is updated shortly after transactions are updated. These tables are also versioned so that concurrent reads and writes are supported with reduced locking.

Support for Memory-Optimized Tables

A non-durable, memory only table is also supported – but in case of a crash all data in these tables will be lost. This feature can improve performance substantially but there is a small window where a crash may cause data loss. This feature is only available in Enterprise edition of SQL Server 2014 and require a 64 bit installation of SQL Server.

In-memory OLTP Engine

SQL Server 2014 includes the new in-memory OLTP engine - an in memory engine that facilitates faster application performance. If your application is facing serious concurrency issues with thousands of simultaneous connections trying to lock data, Hekaton offers an intriguing solution. This feature is arguably getting the most attention from SQL Server watchers, as it promises to drastically speed up SQL Server response times for frequently-accessed data, also known in database parlance as "hot data."

Demo

Database Backup Enhancements

You can now encrypt your backup by specifying the encryption algorithm of your choice. The encryption algorithms supported include: (AES) 128, AES 192, AES 256, and Triple DES.

Buffer Pool Extensions

In previous versions of SQL Server the easy way to fix IO issues was to add memory. This was easy and effective but not always practical and often expensive. Buffer pool extension allows us to extend the buffer pool to storage. SQL Server 2014 will now allow you to extend the memory SQL Server uses by using nonvolatile random access memory (aka Solid State Drives) to greatly improve I/O throughput.

Buffer Pool Extensions

Buffer Pool Extensions provides the seamless integration of solid-state drives (SSD) as a nonvolatile random access memory (NvRAM) extension to the Database Engine buffer pool to significantly improve I/O throughput. Because SSDs has much better IO performance than normal disk we get the benefits of a very large buffer pool without adding more memory.

Buffer Pool Extensions

This feature requires a 64 bit installation and works with Enterprise or Standard edition of SQL Server 2014. The benefits that you get from BFE include: Increased transaction throughput Reduced I/O latency Better read performance

Delayed Transaction Durability

SQL Server 2014 introduces the ability to reduce latency by designating some or all transactions as delayed durable. A delayed durable transaction returns control to the client before the transaction log record is written to disk. Durability can be controlled at the database level, COMMIT level, or ATOMIC block level. This reduces the contention while writing to the transaction log for asynchronous transactions. The benefits include: lesser time for transaction commits, reduced contention and faster release of transaction locks.

Improved Cardinality Estimation

Improved cardinality estimation is a behind the scenes process that improves SQL Server’s ability to create good execution plans. Better execution plans mean improved performance. This feature is supported on all editions of SQL Server 2014.

Natively compiled SPs

Natively compiled SPs are Transact-SQL stored procedures compiled to native code that access memory-optimized tables. Natively compiled SPs allow for efficient execution of queries and business logic in the stored procedure. Natively compiled SPs do not implement the full Transact-SQL programmability and query surface area. There are certain Transact-SQL constructs that cannot be used inside natively compiled SPs. Natively compiled SPs are created using CREATE PROCEDURE (Transact-SQL).

Natively compiled SPs

A natively compiled SP in Hekaton, will be loaded in memory when first created and, subsequently, at first usage after database start-up. Once in memory, it will stay loaded until the database will be taken offline or deleted since there is no way to purge. Additionally, as you probably already know, ALTER is not possible, you have to drop and re-create.

Clustered ColumnStore Indexes

Columnstore indexes are no longer read-only. This allows ETL processes for data warehouses to write, update and delete records during processing. Additional compression of Columnstore indexes is now available by using Alter Index Rebuild with the COLUMNSTORE_ARCHIVE option. The use of Columnstore indexes requires Enterprise Edition of SQL Server 2014.

Clustering Support

An exciting new feature in SQL Server 2014 is the support for the deployment of a Failover Cluster Instance (FCI) with Cluster Shared Volumes (CSV). All nodes in a failover cluster should be of the same architecture - either 32-bit or 64-bit and they should have the same OS version running in them.

Contact:- joydipkanjilal@yahoo.com www.joydipkanjilal.com http://www.infoworld.com/blog/microsoft-coder