sql server 2014 in memory
DESCRIPTION
Slides from my presentation at Code Camp NYCTRANSCRIPT
SQL SERVER 2014 – IN MEMORY OLTP / XTP
(PREVIOUSLY HEKATON)
Ravi Okade
Agenda
Why do you need it ? How it is done Benefits Limitations
Why do you need a new architecture ? In-Memory
Memory is cheapServers have HUGE memorySome data is hotTraditional page based architecture has
limitations, even when all pages are in memory Native Compilation, Lock-less architecture
Individual cores are not getting any fasterWhile number of cores is increasing, Parallel
processing has its limits due to lock contentionYou cannot scale linearly by adding more cores
Comparable Products and Technologies Similar Products
Oracle TimesTenSybase ASE
Other technologies with some similaritiesDistributed Cache
○ Oracle Coherence is one of many distributed cache’s that support query language (CQL). It can also persist to Oracle. Another example with similar capabilities is Gigaspaces
NoSQL Databases○ Most NoSQL databases support distributed querying
and aggressively cache data (e.g. MongoDB)
What it all means..
Use Cases
Hot tables – frequent inserts and updates, but queried heavily e.g:Stock Trading applicationsPatient Vital StatsFlight information (live)
Transient data e.g.:Web Session dataStock Market data
Need for High speed OLTP queries
In-memory Architecture Main Concepts
In-memoryLock free (row versions)Native compiled stored procedures
Data storage optimizationsMemory optimized data structuresTransaction log optimization (block writes, no undo)Data file Optimization (Sequential writes, Merging) Index optimization (Only in-memory, Not persisted to disk,
no tx logging)No TempDBGarbage collection optimization (huh?)
Non-durable option (Not persisted to disk)
In Memory – Myths and Realities
Myth: In-Memory OLTP is the same as DBCC PINTABLEReality: In-Memory OLTP uses a completely
new design built from the ground up. Myth: If SQL Server crashes all data is
lostReality: No – it is persisted to disk and is
fully recoverable.
SQL Server 2014 In-Memory Architecture
SQL Server Integration
• Same manageability, administration & development experience
• Integrated queries & transactions
• Integrated HA and backup/restore
Main-Memory Optimized
• Optimized for in-memory data
• Indexes (hash and range) exist only in memory
• No buffer pool, B-trees• Stream-based storage
T-SQL Compiled to Machine Code
• T-SQL compiled to machine code via C code generator and VC
• Invoking a procedure is just a DLL entry-point
• Aggressive optimizations @ compile-time
High Concurrency
• Multi-version optimistic concurrency control with full ACID support
• Core engine uses lock-free algorithms
• No lock manager, latches or spinlocks
In-Memory integration with SQL Server
Memory Management
Table data resides in memory at all times. No paging. Must configure SQL box with sufficient memory to store
memory-optimized tables. Max supported 512GB Failure to allocate memory will fail transactional workload at
run-time
In-Memory Data Structures Rows
New row format ○ Structure of the row is optimized for in-memory residency
and access.
One copy of row○ Indexes point to rows, they do not duplicate them.
IndexesHash Index for equality search memory-optimized B-tree for range and equality search
(In CTP2)Do not exist on disk – recreated during recovery.
In Memory – Table DDL: Need filestream filegroupCREATE DATABASE Hekaton_DB ON PRIMARY (
NAME = [Hekaton_DB_PRIMARY], FILENAME = 'C:\Hekaton_test\sqlservr\data\
Hekaton_DB_data.mdf'), FILEGROUP [Hekaton_DB_hk_fs_fg]
CONTAINS MEMORY_OPTIMIZED_DATA (NAME = [Hekaton_DB_hk_fs_dir],
FILENAME = 'C:\Hekaton_test\sqlservr\data\Hekaton_DB_hk_fs_dir') LOG ON (NAME = [Hekaton_DB_log], FILENAME='C:\Hekaton_test\sqlservr\data\Hekaton_DB.ldf',
SIZE=100MB)
Table creation internals
CREATE TABLE DDL
Code generation and compilation
Table DLL produced
Table DLL loaded
Table Storage
Filestream is the underlying storage mechanism
Data filesStore inserted recordsData written only upon Tx commit
Delta filesStore deleted records (updates are an
insert/delete pair)
Transaction Logging
Uses SQL transaction log to store content
All logging is logicalNo log records for physical structure
modifications.No index-specific / index-maintenance log
records.No UNDO information is logged
Native Compiled Stored Procedures Compiled to C language and compiled
into a dll using VC Optimized aggressively at compile time Can only access In-memory tables Not all T-SQL constructs and functions
supported No alter procedure – must drop and
recreate
Procedure Creation
CREATE PROC DDL
Query optimization
Code generation and compilation
Procedure DLL produced
Procedure DLL loaded
Using Row Versions for Lock free architecture SQL Server 2014 In Memory DB has no
locks (period). Row versions are used to maintain
updates No TempDB Row Versions which are no longer
referenced are garbage collected. Supports Snapshot, Repeatable Read
and Serializable Isolation levels
Transaction Isolation Levels SNAPSHOT
Reads are consistent as of start of the transactionWrites are always consistent
REPEATABLE READRead operations yield same row versions if repeated
at commit time
SERIALIZABLETransaction is executed as if there are no concurrent
transactions – all actions happen at a single serialization point (commit time)
Garbage Collection
Row VersionsUpdates, deletes, and aborted insert operations create row versions that (eventually) are no longer visible to any transaction.Slow down scans of index structuresCreate unused memory that needs to be reclaimed (i.e. Garbage Collected)
Garbage Collection (GC)Analogous to version store cleanup task for disk-based tables to support Read Committed Snapshot (RCSI)System maintains ‘oldest active transaction’ hint
Design Goals:
Non-blocking, Cooperative, Efficient, Responsive, Scalable
Active transactions work cooperatively and pick up parts of GC work
A dedicated system thread to do GC
Benefits Uses Commodity Hardware Works seamlessly with current SQL Server objects Yes, it is still ACID compliant Yes, you can mix in-memory and disk based tables in
the same database Yes, your transactions can span in-memory and disk
based tables No, you cannot have partial in-memory tables Yes, it has to fit in-memory 100%, forever. Yes, you can limit how much memory is used by the
in-memory tables (think resource pools) (not in CTP1) Yes, you can have high availability
Limitations Row Sizes can’t be larger than 8060 bytes (incl. Variable
Length Columns) LOB, XML Data Types are not supported No Foreign Key and Check Constraints No IDENTITY, SEQUENCE No DML Triggers No ALTER Table (Need to recreate table) No Add/Remove Index (Need to recreate table) No diff backups Indexes are rebuilt (consider startup time) Disk files are merged while being loaded Running out of memory
CTP1 Limitations
No B-Tree index (used for range search) No Resource Pools No always on
Demo Performance against disk based table Performance with native sproc Number of Tx records written disk vs memory Northwind example [TBD] Dealing with transactions [TBD] Performance of T-SQL/Interop sprocs
with In memory tables [TBD] Disk files and merging ? [TBD] Backup, Tx log backup; no diff backups
More information Tech-ed SQL Server 2014 Videos
Microsoft SQL Server In-Memory OLTP: Overview of Project “Hekaton” Microsoft SQL Server In-Memory OLTP Project “Hekaton”: App Dev Deep Dive Microsoft SQL Server In-Memory OLTP Project “Hekaton”: Management Deep Dive Microsoft SQL Server 2014 In-Memory OLTP: Overview SQL Server In-Memory OLTP: DB Developer Deep Dive SQL Server In-Memory OLTP: DBA Deep Dive
SIGMOD 2013 article by Microsoft Hekaton Whitepaper by Kalen Delaney High-Performance Concurrency Control Mechanisms for Main-Memory
Databases, Microsoft Research SQL Server 2014 - MSDN Online Documentation Blog articles by Bob Beauchemin Getting Started with SQL Server 2014 In-Memory OLTP (SQL Server Team blog
) Additional links Oracle TimesTen FAQs SQL Server In Memory Set up and Demos (My Blog)