brainshare 2010 slc - els306 linux disaster recovery made easy
TRANSCRIPT
Linux Disaster Recovery Made Easy
Schlomo SchapiroSystems Architect / Open Source Evangelist
Immobilien Scout [email protected]
Jeff LindholmTechnology Sales Specialist – Linux
Novell, [email protected]
License: http://creativecommons.org/licenses/by-nc-nd/3.0/
© Novell, Inc. All rights reserved.2
Agenda
Linux and disaster recovery
Solutions: relax and recover
Secure disaster recovery for enterprise environments
SLES 11 SP1 HA – ReaR availability and support
Live demo and use cases
Questions and answers
© Novell, Inc. All rights reserved.3
Backup != Restore/Recovery
Linux and Disaster Recovery
© Novell, Inc. All rights reserved.5
Disaster Recovery Strategies
Disk Imaging
+ Offline -> no open file issues
+ Simple to restore
– Online -> very problematic
– No “perfect” open source tool for Linux available
Copy Files and Store Disk Layout and Boot Info
+ Online -> no problem
+ Backup independent of disk layout and sizes
– More effort required to restore (can be scripted)
– Maybe consistency problems, but should be solved by backup solution
© Novell, Inc. All rights reserved.6
Advantages of a Linux System
All information is stored in files; all files always readable
Operation system and applications can be “slim” - 600MB enough for complete standard server
Open system – open methods and procedures
All steps of an installation can be scripts: partitioning, file systems, boot loader etc.
© Novell, Inc. All rights reserved.7
Disaster Recovery – Media
• Most important: external storage!
• Bootable media: CD/DVD, USB key, LAN, tape ...
• Media usually combination boot and backup media:
– Bootable CD/DVD, USB key with backup data on it
– LAN boot (PXE) with backup data via CIFS, NFS ...
– Bootable tapes - HP OBDR (CD emulation)
• Separation between boot media and backup data
– Boot the system from a (small) USB key, CD/DVD or LAN
– Recover the system with backup software, tar, rsync ...
© Novell, Inc. All rights reserved.8
Disaster Recovery – How It Works
• Store the disk layout
– Partitioning, LVM and RAID configuration
– File systems, file system labels ...
– Boot loader (GRUB, LILO, ELILO)
• Store the files (tgz, rsync, through backup software ...)
• Create bootable rescue media with system configuration (and backup data)
• Can be done online
– No business interruption
– 100% compatible with original systems hard- and software
© Novell, Inc. All rights reserved.9
Disaster Recovery – Rescue Media
• Create “rescue linux” from running system
• Optimally compatible “tool box”
• Clone the system environment
– Linux kernel and modules
– Device driver configuration
– Network configuration
– Basic system software and tools
• Operate entirely in RAM (initrd)
© Novell, Inc. All rights reserved.10
Disaster Recovery – In Action
• Boot system from rescue media
• Restore disk layout
– Create partitions, RAID configuration and LVM
– Create file systems (mkfs, mkswap)
– Configure file systems (labels, mount points)
• Restore the backup data
• Restore the boot loader
• Reboot
• Done!
Solutions: Relax and Recover
© Novell, Inc. All rights reserved.12
Proprietary Solutions
• Usually part of backup software
• Extra licenses and can be costly (up to 1000€ / server)
• Some are free, but are the any good?
• Level of integration into backup software varies
• Not all backup software actually supports disaster recovery
• Cover only simple setups
• Often static boot media, inflexible with regard to drivers
• Certification on server hardware missing or lagging behind procurement cycles
© Novell, Inc. All rights reserved.13
Open Source Solutions
• Several software projects on the internet:
– Relax and recover
– mkCDrec
– Mondo Rescue
– CRU
– BACULA
– Your-Favorite-Selfmade-Script
– <something you know but I never heard about>
• In 2006, all of them where lacking something...
© Novell, Inc. All rights reserved.14
Relax and Recover – ReaR
http://rear.sourceforge.net
• GPL Software – Developers in Germany and Belgium
• 100% Bash script – no GUI and no dependencies
• Utilize kernel, modules, binaries of host (kernel ≥ 2.6)
• Support any combination of SW/HW RAID, LVM
• Internal backup on CIFS, NFS ...
• Boot media on CD/DVD, USB key and LAN (PXE)
• Successor of mkCDrec
© Novell, Inc. All rights reserved.15
ReaR – Features
• Focus on disaster recovery and not backup
• Tight integration with common backup software – delegate file backup to backup infrastructure
• Simple full backup integrated
• Complements backup software:
– Backup software: Data storage and retrieval
– ReaR: Recover system layout and make it work again
– ReaR utilizes the backup software to restore the backup data
• Use the best tool for the job
© Novell, Inc. All rights reserved.16
ReaR – Backup Software
• Supported solutions include:
– CommVault Galaxy
– IBM Tivoli Storage Manager
– Veritas NetBackup
– HP Data Protector
– Rsync and other “external” methods
– tar.gz archive on NAS share – CIFS, NFS, NCP ...
• Very transparent integration
• Quick implementation: 2-3 PT programming
• Can be easily extended to support other vendors
© Novell, Inc. All rights reserved.17
ReaR – Network Integration
• Disaster recovery as part of network infrastructure– Backup software – file-level backup storage
– ReaR – system environment
– Boot rescue media via PXE – no physical media required
– Very scalable – automated installation of entire disaster recovery data center
BackupStorageNetwork
infrastructure
SoftwareBackup
ReaRBoot Files
PXE Boot
© Novell, Inc. All rights reserved.18
ReaR – Status
• Stable software
– i386 and x86_64 well tested
– ia64 and ppc experimental
• Regular releases (RPM, DEB, TGZ)
• Major Linux distributions ship ReaR:
– SUSE® Linux Enterprise High Availability Extension 11 SP1
– openSUSE® 11.2 and Fedora 11
• Community and commercial support available
• Regular patch submissions from ReaR users
© Novell, Inc. All rights reserved.19
ReaR - Development
• Open Source development model:– Submit patches and feedback – “field testing”
– Sponsoring
• Modular design:– ReaR is a framework to plug together many small bash scripts
– Maximize code reusability
– Simple development model (vi works fine)
– Little to no “interferences” between different areas of code
• Documentation on project homepage– Hierarchical structure
– References to source code
© Novell, Inc. All rights reserved.20
ReaR – Configuration
• Shell-script style configuration file
• Usually: Set backup software and boot media
System definition: ARCH = Linux-x86_64 OS = GNU/Linux OS_VENDOR = SUSE_LINUX OS_VERSION = 11 OS_VENDOR_ARCH = SUSE_LINUX/x86_64 OS_VENDOR_VERSION = SUSE_LINUX/11 OS_VENDOR_VERSION_ARCH = SUSE_LINUX/11/x86_64Configuration tree: Linux-x86_64.conf : OK GNU/Linux.conf : OK SUSE_LINUX.conf : missing/empty SUSE_LINUX/x86_64.conf : missing/empty SUSE_LINUX/11.conf : missing/empty SUSE_LINUX/11/x86_64.conf : missing/empty site.conf : OK local.conf : OK
© Novell, Inc. All rights reserved.21
ReaR – Operation
rear [Options] <command> [command options ...]Relax & Recover Version 1.7.23.2 / 2010-03-19Build: b80d32a97e41a6ec8a5b19cd6399e444Copyright (C) 2006-2009 Schlomo Schapiro Gratien D'haese, IT3 ConsultantsRelax & Recover comes with ABSOLUTELY NO WARRANTY; for details see the GNU General Public License at http://www.gnu.org/licenses/gpl.html
Available Options:-V version information-d debug mode-D debugscript mode-S Step-by-step mode-s Simulation mode (shows the scripts included)-q Quiet mode-r a.b.c-xx-yy kernel version to use (current: 2.6.32.8-0.3-default)
© Novell, Inc. All rights reserved.22
ReaR – Operation (continued)
List of commands:dump Dump configuration and system informationhelp print out usagemkbackup Create rescue media and backup system.mkbackuponly Backup system without creating a (new) rescue media.mkdeb Create DEB packages with this rear versionmkdist Create distribution tar archive with this rear versionmkrescue Create rescue media onlymkrpm Create RPM packages with this rear versionmktar Create tar archive with this rear installationmkvendorrpm Create vendor RPM with this rear versionrecover Recover the systemvalidate Submit validation information
© Novell, Inc. All rights reserved.23
ReaR – Example Configuration
# cat /etc/rear/local.conf
OUTPUT=ISO
#BACKUP=TSM|NBU|DP|GALAXY|EXTERNAL|...
BACKUP=NETFS
NETFS_URL=nfs://server.domain/backup-path
MODULES_LOAD=( vmxnet )
© Novell, Inc. All rights reserved.24
ReaR – Example Execution
# rear mkbackup
Relax & Recover Version 1.7.23.2 / 2010-03-19
The preparation phase OK
Physical devices that will be recovered: /dev/sda /dev/sdb
Creating root FS layout OK
Copy files and directories OK
Copy program files & libraries OK
Copy kernel modules OK
Create initramfs OK
Creating archive 'nfs://server.domain/backup-path/rear/backup.tar.gz'
Transferred 297 MB in 254 seconds [1197 KB/sec]
Making ISO image OK
Wrote ISO Image /tmp/ReaR.iso (19M)
Copying resulting files to network location OK
The cleanup phase OK
Finished in 258 seconds.
© Novell, Inc. All rights reserved.25
ReaR – Disk Layout
# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/system-root
2,5G 927M 1,4G 40% /
devtmpfs 244M 128K 244M 1% /dev
tmpfs 247M 0 247M 0% /dev/shm
/dev/sda1 84M 19M 61M 24% /boot
/dev/mapper/system-home
524M 17M 481M 4% /home
/dev/md0 680M 55M 591M 9% /var
© Novell, Inc. All rights reserved.26
ReaR – Disk Layout
# pvs PV VG Fmt Attr PSize PFree /dev/sda3 system lvm2 a- 3,21G 0 /dev/sdb2 system lvm2 a- 312,00M 0
# lvs LV VG Attr LSize Origin Snap% home system -wi-ao 532,00M root system -wi-ao 2,44G swap system -wi-ao 572,00M
# cat /proc/mdstat Personalities : [linear] [raid1] md0 : active raid1 sda2[0] sdb1[1] 706816 blocks super 1.0 [2/2] [UU]
© Novell, Inc. All rights reserved.27
ReaR – Recovery Example
# rear recoverRelax & Recover Version 1.7.23.2 / 2010-03-19Backup archive size is 298MInitializing physical devices OKCreating Software RAID devices OKCreating the LVM2 devices OKCreating file systems OKRecreated this filesystem layout:Filesystem Size Used Avail Use% Mounted on/dev/mapper/system-root 2.5G 68M 2.3G 3% /mnt/local/dev/sda1 84M 5.6M 74M 7% /mnt/local/boot/dev/mapper/system-home 524M 17M 481M 4% /mnt/local/home/dev/md0 680M 17M 629M 3% /mnt/local/varRestoring from 'nfs://laptop/dev/shm/nfs/rear/backup.tar.gz' OK
© Novell, Inc. All rights reserved.28
ReaR – Recovery Example (continued)
Restoring the bootloader (SuSE style)
Creating swap files and partitions OK
Finished recovering your system
You can explore it under /mnt/local
Thank you for using Relax & Recover Version 1.7.23.2
Finished in 150 seconds.
© Novell, Inc. All rights reserved.29
ReaR – Vision
• Do more than just disaster recovery– System cloning and provisioning
– System migration (P2V, V2P, V2V, P2P)
– Generic system tool-kit for offline tasks and rescue media
• Centralized information and management:– rear-server
– 1.0: Gather and aggregate information about ReaR
> System protection status
> Rescue media boot files
– 2.0: Centrally manage configuration and control ReaR
> Site-wide and system-local configuration store
> Trigger and control disaster recovery tasks
– 3.0: Integrate into 3rd party solutions
Secure Disaster RecoveryEnterprise Environments
© Novell, Inc. All rights reserved.31
Enterprise DR – Requirements
Standardization
Scalability
Automation
Integration in existing system environment
Security
Coverage over all systems in use
Support and maintenance
Education and certification
Affordable licensing – site license
© Novell, Inc. All rights reserved.32
Enterprise DR – Relax and Recover
• Standardization through global configuration:
– /etc/rear/site.conf
• Scalability and automation:
– rear mkrescue can be run as a cron job
– Automated processing of rescue boot media fairly simple
– Rescue boot media also stored with backup software
– LAN boot – extreme scalability and automated DR possible
• Integration in existing system environment
– Support internal and external backup solutions
– E-mail reports
© Novell, Inc. All rights reserved.33
Enterprise DR – Relax and Recover
• Security considerations– No “secret” data in rescue media
– Secure export of rescue media through backup software and E-mail
– Remote access to rescue system only with SSH keys
• Coverage– i386 and x86_64 stable
– Enterprise Linux (Novell®, Red Hat, Ubuntu) stable
– Community distributions also work, less tested
• Maintenance and support, education– Please contact Novell and the authors
• Site license: Included with GPL
© Novell, Inc. All rights reserved.34
Security Aspects – Disaster Recovery
Transfer of Backup Data
• How secure is the backup software?
• Insecure: NFS2, NFS3
• Secure (with authentication):
– NFS4, CIFS
– SSH (rsync)
Transfer of Boot Riles
• Insecure (anonymous):– NFS
– CIFS
– SSH (keys)
– RCP
• Manipulation proof:– SMTP
DR client
Backup server DR boot server
© Novell, Inc. All rights reserved.35
Enterprise Linux Disaster Recovery
• Linux – Simple Disaster Recovery
• Integration: Disaster recovery – backup software
• Automated creation of disaster recovery information
• Network boot into rescue system
• Disaster recovery becomes part of infrastructure
• Scales well even for very large data centers
• Disaster recovery becomes a valid option for clients, not only for servers
SUSE® Linux Enterprise High Availability Extension 11 Service Pack 11 –
ReaR Availability and Support
© Novell, Inc. All rights reserved.37
SUSE® Linux EnterpriseModular Portfolio
SUSE Linux Enterprise
Custom OS
DeveloperTools
Bundles
SUSE Linux Enterprise
ServerJeOS SLE SDK SLE POS
SLES for SAP,Expanded Support
SLED
Server OS
DesktopOS
SupportSKUs
High Availability Real Time Extension Mono Extension
Optional extensions for high availability clustering, deterministic computing, and running .NET applications
Pay only forwhat you need
Common CodeBase foundation
Level 3 Support Runs on x86, x86_64, Itanium,
Power, and s390x
© Novell, Inc. All rights reserved.38
SUSE® Linux Enterprise High Availability Extension 11
Key Features
• Flexible, policy driven clustering solution
– OpenAIS messaging and membership layer
– Pacemaker cluster resource manager
• Cluster-aware file system and volume manager
– Oracle Cluster File System (OCFS2)
– Clustered Logical Volume Manager (cLVM2)
• Bare Metal Recovery (ReaR)
• Continuous data replication
– Distributed Replicated Block Device (DRBD8)
• GUI and CLI tools– Graphical user interface
– Unified command line interface
• IP load balancing
• Resource agents
• Virtualization aware
© Novell, Inc. All rights reserved.39
SUSE® Linux Enterprise High Availability Extension 11
Differentiators
• AffordableCosts less than other Linux clustering solutions, with tools and resource agents included at no extra charge
• Virtualization-awareSupports hybrid virtual and physical clusters
• Open SourceNo proprietary modules or user space applications means your Linux clustering solution is secure and easily kept up to date
• IntegratedStreamlined support and an optimized solution from a single vendor that delivers both the OS and clustering software
© Novell, Inc. All rights reserved.40
SUSE® Linux Enterprise High Availability Extension 11
New Features/Enhancements SP1
ReaR – Relax and Recover
• ReaR will be included with SUSE Linux Entreprise Server 11 SP1 HA Add-on
• ReaR is considered to be an extensible framework, allowing for custom scripting and extension of the recovery toolset
• ReaR RPM packages will be supported on x86 and x86_64
• SLE11 High Availability Add-on inherits service level agreement from the SLES Subscription
– Priority (24x7)
– Standard (12x5)
• Customization is not covered by Novell® Technical Support
– Integration Partners are available (contact Novell Sales)
•
© Novell, Inc. All rights reserved.41
SUSE® Linux Enterprise High Availability Extension 11
More Information - SP1
For more information on what is new with SLE11 HA SP1:
ELS307 Building High Availability Clusters with SUSE Linux Enterprise High Availability Extension
ELS302 SUSE Linux Enterprise High Availability Extension 11: Support and Troubleshooting
ATT307 Quickly Configure a High Availability Cluster with SUSE Linux Enterprise 11 High Availability Extension
Live Demo/Use Cases
© Novell, Inc. All rights reserved.43
Relax and Recover – Live Demo
Demo (~10 min):
• SUSE® Linux Enterprise High Availability Extension 11 SP1
• ReaR creates backup and boot CD
• “PEBKAC” at work
• Boot system from CD
• Recover system
• Happy End
Questions and Answers
Unpublished Work of Novell, Inc. All Rights Reserved.This work is an unpublished work and contains confidential, proprietary, and trade secret information of Novell, Inc. Access to this work is restricted to Novell employees who have a need to know to perform tasks within the scope of their assignments. No part of this work may be practiced, performed, copied, distributed, revised, modified, translated, abridged, condensed, expanded, collected, or adapted without the prior written consent of Novell, Inc. Any use or exploitation of this work without authorization could subject the perpetrator to criminal and civil liability.
General DisclaimerThis document is not to be construed as a promise by any participating company to develop, deliver, or market a product. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. Novell, Inc. makes no representations or warranties with respect to the contents of this document, and specifically disclaims any express or implied warranties of merchantability or fitness for any particular purpose. The development, release, and timing of features or functionality described for Novell products remains at the sole discretion of Novell. Further, Novell, Inc. reserves the right to revise this document and to make changes to its content, at any time, without obligation to notify any person or entity of such revisions or changes. All Novell marks referenced in this presentation are trademarks or registered trademarks of Novell, Inc. in the United States and other countries. All third-party trademarks are the property of their respective owners.