To dedup or not to dedup ?

Deduplication Deep Dive

About Me

Kamil Bączyk

• Core & BPOS Consultant \ Trainer • MCT, MCSE, MCSA, MCITP, MCTS, MCP, ITIL, CEHv7, etc ….

• @ Facebook, LinkedIn, Goldenline, Xing …• baczyk.kamil@gmail.com

Time to talk • Facts• Discuss Windows Server 2012

Disk Deduplication • Demo

FactsBoiling Points:• File Data Growth

Data demand rises Increased stress – backup solutions

• Storage TCO Disk cost Data Management

• Consolidation• Data decentralization

Offline and secure data WAN / LAN Access IDC Worldwide File-Based Storage

Forecast, doc #231910, 2011

Facts – what is it ?

• Goal :use less storage • Method :Check and ensure that identical content in multiple files (big) is only stored once• Block-based, post-process,

transparent solution

Dedup

Facts – what is it ?MODELS:• Source :Prevent transferring data, if duplicate• Inline :Perform dedup when data is written (NTFS Compression, Slow• Post-Process :Works after, in the background, Used in Windows Server 2012

Dedup

Facts – what is it ?Other ways:• SIS (Single-Instance-Storage):File based solution in W2K• NTFS Compression :Inline, CPU based• NTFS hard links :Not transparent, file-based

Dedup

Facts – how it works ?

• Segment data into „chunks”• Identify duplicate chunks• Maintain a single copy of each

„chunk”• Compress• Reference existing files with

reparse points • Default (5day policy)• Free

Facts – how it works ?• Manage by PowerShell, WMI, Server Manager,• Transparent for data – Mini Filter• Maintain a single copy of each „chunk”• Compress Data Chunks• Dedicated Service (DDPSVC)• Default (5day policy)• Audit - Event Viewer\Applications and

Services Logs\Microsoft\Windows\Deduplication\

• Chunks files 32-64 KB• CRC Checking – Hot Spots

Facts – how it works ?

• Only on Windows Server 2012• Filter Driver• Windows Cache – Dedup Aware• No GPO• Doesn’t work on CSV but

supports cluster data • No support for Boot Drives or

System Drives

• Works on NTFS, no ReFS support• Excluded specified files by default

or manually• DDPSVC – checks 100Gb ~ 1 hour• Default policy 5days • Scheduled process• Live SQL DBs or Exchange • Live Production VMs / VDI

Discussion ?When to use it ?• VHD backup library• Folder redirection servers

• DB backup volumes• Software deployment shares

Discussion ?How to check ?Dedup Eval tool PowerShell ?

Demo

Deduplication Deep Dive

Q & A

Thank You

Kamil Bączyk

• Core & BPOS Consultant \ Trainer • MCT, MCSE, MCSA, MCITP, MCTS, MCP, ITIL, CEHv7, etc ….

• @ Facebook, LinkedIn, Goldenline, Xing …• baczyk.kamil@gmail.com