Wednesday 2nd February 2011

Tristan Self – Senior IT Infrastructure Engineer – Oaklands College

Who am I? Contents of Presentation

• Introduction• Overview of DPM 2007• Data and System Protection• How DPM Works• Oaklands College Case Study – Old Backup System and Problems• Oaklands College Case Study - How DPM 2007 has Solved These Problems• Oaklands College Case Study - Hardware Required and DPM Configuration• Why DPM fits FE Organisations?• What problems with DPM• DPM Implementation Procedure• DPM in Production• Licensing/Costs• Disaster Recovery• What’s New in DPM 2010• Books/courses• Conclusion• Q and A

Protecting data and systems comprehensively is difficult especially when systems become large and budgets and staff are stretched.

Oaklands College has worked to the following 3 Tier Data and System Protection Strategy of which Microsoft DPM is a key part.

3 Tier Data and System Protection Strategy

Microsoft Solution to Data Backup and Restore Data (2007 - 2nd Gen. Product) Allows Backup and Restore of all Core Microsoft Products:

◦ System State (Active Directory) – Windows Server 2003 and 2008◦ Files/Folders - Windows Server 2003 and 2008 (and XP/Vista/7)◦ SQL Server 2000 SP4, 2005 SP1+ and 2008 (& Clustered)◦ Exchange Server 2003 SP2, 2007 and 2010 (& Clustered)◦ SharePoint 2007 and 2010◦ Hyper-V Virtual Machines (rudimentary)

Utilises VSS (Volume Shadow Service) in Windows Server Disk to Disk (to Tape) and Disk to Tape Designed for SMB to Enterprise Organisations with Short Backup Windows

and Large Amounts of Data. (Like a College with 1000’s GB of data) Efficient – “De-duplication” of Disk Backups and “Throttleable” bandwidth

and minimal client utilisation (disk/cpu). Fast Recovery from Disk to Production Server Without Need for Tapes Continuous Data Protection (Can be Synchronised as often as 15 Minutes) Multiple “Full Backups” per Day!

Typical Microsoft DPM Infrastructure

A protection agent is installed on any server with data requiring protection such as a file server, Exchange or SQL server.

When directed by the DPM server this agent is responsible for triggering the VSS snapshot and moving “Synchronising” the data from the protected server to the DPM server.

Disk to DiskDisk to Disk to TapeDisk to Tape

The data is stored in the DPM storage pool as a “replica” of the volume and a collection of “recovery points” (changed versions of the replica at a given point in time.)

Aged recovery points are copied to tape based on the configuration of the retention.

Storage Pool

The agent is installed on the protected server, the files and folders selected for backup, then DPM creates a replica of the data (full backup.)

The agent then monitors the protected data (blocks) for changes and synchronises only the changed blocks to the DPM server at regular intervals.

At specified time the DPM server generates a “recovery point” (a consistent full backup of the data) on the replica volume.

Only the changed blocks between the new recovery point and the previous recovery point are stored (on the recovery point volume.)

Replica Volume

Recovery Point Volume

This is the basic principle to which DPM operates irrespective of data type being protected.

Express Full Backup – Generally Performed once a day. (But can be more if you use a “Simple” backup mode (e.g. SQL Server), you can use these to create multiple full backups per day.)

Transaction Log Backup - In addition to an Express Full Backup, at a specified interval, e.g. every 15 minutes—DPM pulls and stores the transaction logs of the application. ◦ In the event of a recovery, DPM restores the last express full backup, then

applies all the transaction logs created since that backup was made. ◦ If only the database disk was corrupt and the transaction logs were on a

separate disk on the live machine, DPM would also play back any transaction logs that were still on the live server that had not yet made it to DPM.

◦ This process means “zero loss” for your transactional applications, assuming the database was corrupt, BUT....not your server was a melted mass of metal and plastic, in this event you would have data up to the point of the last synch of the transaction logs...

Only Changed Blocks - Again like with a file/folder backup only the changed blocks are “synched” with the DPM server; reducing server and network load.

Restore Granularity

For individual Exchange mail item restores the database can be mounted offline with an EDB file opening tool and then items exported to PST for Exmerging or importing back into the mailbox.

FE College based in St. Albans and WGC in Hertfordshire, about 850 staff and 13,000 students.

5 campuses all with servers and data to be protected, but only 3 campuses are permanently manned with IT staff.

Busy inter-campus wireless site links History of data loss and IT failures, complete loss of all e-mail in 2007, users lacking

confidence of protection of their data from loss. Wide range of applications and data sources for protection:

◦ System State (Active Directory)◦ SQL Server◦ Exchange Server◦ SharePoint 2007 Server◦ File/Folder Data◦ Oracle◦ MySQL

7TB of file data of staff and students. 300GB+ Exchange Email Data. Numerous SQL Databases and Database Servers.

•75 Servers (Physical + Virtual)•2,500 Workstations

•Microsoft Based (almost completely)•Microsoft Windows 2003/2008

Server

Old System◦ Dell PowerEdge and Dell Tape Library LTO2– Microsoft Windows 2000 Server with Veritas NetBackup

Problems◦ Aging Hardware and Software

Aging hardware and software (6 Years old)◦ Support Intensive

Constant hardware and software failures, as well as jobs freezing or failing part way through. Complicated interface and procedures meaning reliance on a single member of staff.

◦ Bandwidth and Disk Intensive Bandwidth and disk intensive, backups could often not be re-run soon after failing due to the

performance impact of running them during the day.◦ Incomplete Backup

Not all data captured due to disk and tape space limitations. “Monolithic” backups of SQL databases and Exchange data using NTBackup constantly failing. Full backups of main college file server 1.4TB barely able to finish over a weekend, liable to failure.

◦ Limited Support for Applications No direct support for SQL, Exchange or SharePoint, relied on the “disk dump and copy” method to

capture a backup. This often failed due to the data size and was fiddly to restore in an emergency.◦ Restores Unreliable and Difficult

Restores required input of tapes where unreliable and could take many hours for even a single file to be found and restored.

Not able to meet recovery point objective of a daily backup of all college data.◦ No Disaster Recovery Support

Requirements of New Backup System◦ Minimise Cost

To be as cheap as possible for hardware and software licensing.

◦ Minimise Support Minimal day to day support required. Simple interface so more staff can operate backup system.

◦ Disaster Recovery Enabled Solution◦ Meet “Recovery Point Objective” of a Daily Backup of all College Data◦ Reduce Restore Time

Restoration must be quicker and easier to perform, recent data should not be on tape and should be on disk.

◦ Efficient and Minimal Performance Impact Minimal performance impact to production servers and network when backups/restores are taking place.

◦ Tape Backup Solution Included Should be easy to use and configure and must provide long-term protection running for up to 7 years on

tape.

◦ Application and Infrastructure Support for Microsoft Apps and Virtual Environment Support for backing up data from Windows Servers running in a VMWare virtual infrastructure. Must directly support protection of SQL Server 2000 SP4, 2005 and 2008, Exchange 2003/2007/2010 and

SharePoint 2007 Must give a reasonable solution to allow backup of non-Microsoft apps via an agent or other means.

◦ Scalable Must scale easily to allow for more data to be stored and more servers to be protected.

Aging Hardware and Software◦ New hardware and software utilising inexpensive SATA RAID for disk storage and latest LTO4

tape drive to replace hardware. Support Intensive

◦ Simple GUI and better reliability has allowed a wider range of staff to be able to use the backup system.

◦ Central monitoring of jobs and central tape management. Bandwidth and Disk Intensive

◦ VSS and Changed Block “Synch”– Minimises network bandwidth use during backup/restore.◦ Minimal impact to production server disk activity during backups.

Incomplete Backup◦ De-duplication disk store – All data can be backed up, and stored on disk for many days with

minimised disk usage. Then archived to tape for long-term storage.◦ 3 full backups a day of file servers, 1.6TB server can be backed up in minutes rather than

days using “Express Full Backup” , full Exchange backup every 2 hours, better than the “monolithic” full backup copy required before.

Limited Support for Applications◦ Direct support protection of SQL Server 2000 SP4, 2005 and 2008, Exchange

2003/2007/2010 and SharePoint 2007◦ Oracle and MySQL captured using the “dump to disk” method.

Restores Unreliable and Difficult Recovery Point Objectives – Meet and often exceeded! Restores of files and database data take only minute to find the file and

restore it from disk No Disaster Recovery Support Secondary off-site server support

CPU◦ Dell PowerEdge R710 – Intel Xeon E5530 – 2x CPU 2.4Ghz

RAM (DPM is RAM hungry)◦ 32GB (more if you can afford it)

Server Disks (DPM is IOPS hungry)◦ 2 x 15k SAS HDD - RAID 1 – OS, DPM Binaries and SQL Log Files◦ 4 x 15k SAS HDD - RAID 5/10 – SQL Database Files

Storage Pool Disks (Based on your data size)◦ 2 x Dell MD1000 with 15 x 1TB 7.2k SATA Disks with PERC 6/E

Controller Tape Library (Based on your retention and data volume)

◦ 1 x Dell TL2000 with LTO 4 Tape Drive (24 Tape Library) Microsoft Windows 2008 Server Enterprise (64-Bit) Microsoft SQL Server 2005 Standard (64-bit)

Reasonably Simple to Implement Simple to operate, more staff can be involved in

file/folder restores. Minimal Support Required Supports a large environment with large amounts

of data. Software cost is less than that of similar

competitor products (with Microsoft Educational Licensing.)

When a student deletes all their work you can restore it quickly and easily from disk within minutes.

Anti-Virus Issues – Kaspersky Anti-virus can’t cope with the mount points for the DPM storage pool and locks the volumes if it sees a virus.

Microsoft Only – If you run a Microsoft shop this won’t be an issue, if you have Mac or Unix/Linux servers or non-Microsoft applications this could become problematic.

Design - Relies on good protection group design to make it effective. Changing the design afterwards is difficult and time consuming.

SharePoint - Backup/restore works but is clunky in DPM 2007, but more stable and easier to use in DPM 2010. Requires another SharePoint licence and recovery server for item level restores.

DPMDB Corruption – All of DPM relies on its database, as long as this is backed up you’re ok. However backup of this is tricky without a secondary server to avoid catch 22 situation.

Inconsistent Data Volumes and Manual Intervention – Replica/RP volumes don’t grow automatically, volumes can become inconsistent sometimes. Both these problems are resolved in DPM 2010.

Agree Data Protection Policy with Business◦ “Lose no more than four hours of production data.”◦ “Provide me with a retention range of 30 days.”◦ “Make data available for recovery for seven years.”

Design Protection Group Configuration Design the DPM Backup Infrastructure

◦ Size the disk and tape pools required◦ Check the hardware compatibility List◦ Determine number of licences required

Purchase Hardware and Software Implement DPM server(s) Deploy DPM Client and Create Protection Groups Create First Replicas of all Data Sources Decommission Old Backup System Perform, test and document Restore Procedures

Works fine with Physical or Virtual Machines (VMWare 3.5 & 4 or Microsoft Hyper-V)

DPM server must be Physical Staging Server is recommended for restores of databases,

SharePoint or large file data. No noticeable performance overhead to disk or network with

DPM running. Ensure 15K SCSI disks in RAID 5/10 for DPMDB Disks –

Performance Hungry RAM Hungry (Minimum 32GB RAM) Protection Group Design Considerations Protect the DPMDB at all costs! Use Microsoft 2008 Server R2 Enterprise (64-BIT) for best

performance and scalability. Use local SQL install – Maintain performance and removes

need for additional SQL License

DPM Server Licensing• Microsoft DPM 2007 Server license• No SQL Server license needed if using local DPM SQL install

DPM Client Licensing• DPM Enterprise CAL –

• System State, File/Folder, and SQL, Exchange, Hyper-V and SharePoint• DPM Standard CAL

• System State and File/Folder• Licenses managed from within DPM console

DPM Server OS Licensing• Microsoft Windows 2008 Server R2 Enterprise (64-Bit)

SharePoint Recovery Farm / Recovery Staging Server• Microsoft Windows Server OS License• Microsoft SharePoint 2007 License (1 Server needed only)• Microsoft SQL Server 2005/2008 License

Hardware Costs

Software Costs

Microsoft DPM 2007 Server Licence x 1 = £31.45Microsoft DPM 2007 Enterprise Data Protection ML x 40 = £23.41 x 40 = £936.40Microsoft DPM 2007 Standard Data Protection ML x 20 = £8.53 x 20 = £170.60(If using DPM integrated SQL, no SQL server licence is needed.)

Prices correct as of July 2009 – Microsoft Campus Agreement

Total DPM Software Cost = £1904.25

Dell PowerEdge R710 – Intel Xeon E5530 – 2x CPU 2.4Ghz – 32GB RAM = ~ £7000

Dell PowerVault MD1000 – 15 x 1TB SATA 7.2K HDD = ~£9,000(includes cost of external RAID controller card)

Dell PowerVault TL2000 with LTO-4 Tape Drive = ~£9,500(includes cost of external controller card)

LTO4 Tapes = ~£25 each Total DPM Hardware Cost = £ 25,000

A secondary DPM server can be setup on another campus/site to protect the primary DPM server and its replicas. Off-site backup and Disaster Recovery for your data simply implemented.

This can be a disk only backup, and can have a small disk pool with a shorter retention to save costs.

Failure of the primary DPM server does not affect restore from disk as these can be made from the secondary.

DPM clients can be easily switched to use the secondary DPM server instead.

Backup uses minimal bandwidth as only the changed blocks are sent to the secondary. Can be scheduled and throttled for out of hours.

Performance and Reliability Improvements (Self-Healing) Automatically Grow Volumes as Required Shrink Volumes to use Disk Space Effectively SharePoint 2010 no Longer Requires Recovery Farm Server

for Item Level Restores SharePoint Backups more Complete and Reliable Tape Management Improvements Protect Standalone Servers (non-domain, workgroup or

DMZ) Support for More Data Sources Better, more complete Hyper-V Support Improved Self-Service end users for file and SQL Databases

Mastering System Center Data Protection Manager 2007 - ISBN: 978-0-470-18152-2

Managing Microsoft System Center Data Protection Manager 2007 - Course 50023A

Thank You for Listening

Any Questions?