linux disaster recovery solutions

Linux Disaster Recovery Solutions

IT3 ConsultantsGratien D’haeseGratien D’haese

March 2008

Agenda

• What is Linux Disaster Recovery?• Why are backups not enough?• Commercial or Open Source?• Comparing 3 GPL DR on Debian/ia64

Some Definitions

• What is Disaster Recovery?– The process by which a business function is

restored to the normal, steady state after a disaster

• What is Business Continuity– The way that a business function will operate

after a disaster, until such time as the normal, steady state is restored

What is Linux Disaster Recovery

• Like any other UNIX (including HP-UX) Linux is vulnerable for disaster to strike

• The question really is “What shall I do if a disaster strikes?”

• Dependent on:– HW failure (boot disk lost)– Lost everything (fire, water, earthquake, theft)

• The answer: “Act immediately”

Why are backups not enough?

• Backups of data are necessary!• Are not enough in case of losing the OS!• Reinstalling from scratch takes hours• Fine-tuning of configurations takes days• Even months later issues pop up!• It is absolute necessary to foresee an

inventory of HW and SW

Disaster Recovery Plan (DRP)

• DRP addresses need to recover from an emergency with minimum impact to the enterprise– Protects enterprise from major services failure– Minimizes risk to enterprise from delays in

providing services– Guarantees reliability of standby systems by

testing and simulation– Minimizes personnel decision-making

required during disaster recovery

Commercial or Open Source?

• There is no standard solution delivered with Linux as such

• The choice is do we go for a commercial or Open Source solution?

• Try before you buy I would say– It doesn’t always work as promised– Do we have test equipment? Don’t try it on

production without a real DR test first!

Commercial Solutions (1)

• Storix’ Adaptable System Recovery– www.storix.com– AIX/Linux backup – DR – cloning – HW migration (favors TSM

integration)– Pro:

• Supports IA32 and IA64 platforms• good support with regular updates• V6.2 is able to make bootable USB disk incl. backup

– Contra: • is a backup solution with DR capabilities• GUI interface nice, but useless in DR scenarios• Too tight with TSM• Central server with clients model: administration overhead• Without a valid license no recovery?• Too much choices, hard to remember• Backups are useless without Storix’ software


• Acronis’ True Image Echo Server for Linux– www.acronis.com– Backup/restore software with bare-metal DR

possibilities– Pro:

• Disk image restore• Nice GUI based solution• Good support

– Contra:• Only x86 and x86_64 are supported• Is primarily a backup/restore solution• Console mode is a nightmare• Backups are useless without acronis’ software


• Arkeia Options (of Arkeia Network Backup)– www.arkeia.com– DR from a central backup server– Pro:

• Support• GUI based

– Contra:• Only x86 and x86_64 are supported• Is primarily a backup/restore solution• No DR possible without network (central backup server)• DR scenario is Question/Answer game

Commercial Solutions Summary

• If you decide to go for a commercial solution and you are willing– To pay for it– Wait for updates in case of problems– Need commercial support– Good documentation

• Then go for Storix’ solution

Open Source Solutions (1)

• We have 3 types here– DR optional with Open Source backup software

• Similar comments as with the commercial ones• Completely dependent on backup solution

– Image makers (cloning)• Disk to image or partition to image

– True DR Open Source software• No focus on pure backups (incremental)• Main focus on fast DR• No fancy GUIs can be expected (nobody cares about a GUI

if a disaster strikes)


• Bacula, the network backup tool– www.bacula.org– Is a network based backup/restore program

competing with e.g. TSM, DP, a.o.– Pro:

• Open Source• DR CD-ROM creation based on mkCDrec

– Contra:• DR is a option and runs behind with coding/fixing


• Partimage– www.partimage.org– Saves partitions to an image– Pro:

• Open Source• Image creation can be scheduled across network• Good for exact cloning

– Contra:• Need extra soft (SystemRescueCD) to restore image from

scratch• No explicit DR functionality• No flexibility to other HW


• Clonezilla, a better ghost program– clonezilla.sourceforge.net– Saves partitions to an image– Pro:

• Open Source• Based on partimage• Good for exact cloning across network (to 40 at a time) with

clonezilla server– Contra:

• Image creation done via special live CD, or via clonezilla-server and client program

• No explicit DR functionality• Only for experienced users


• Mondorescue– www.mondorescue.org/– Available since 2000– GPL DR supporting Linux (x86, x86_64, ia64)

and FreeBSD (x86)– Support major Linux distributions– Target backup media includes CDR, DVD-

R(w), USB/disks, tape, network– Understands major file systems including

LVM, soft- and hardware RAID


• Mondorescue– Pro:

• Open Source with excellent support base (incl. HP)• CLI and GUI interface• Cloning supported• Uses a fail-safe Linux kernel (mindi)• Pretty good documentation

– Contra:• Command line quite complicated to remember• Relies on different sub-products (mindi, mindi-busybox, afio,

…)• Lacks integration with other backup solutions


• Make CD-ROM Recovery (mkCDrec)– mkcdrec.ota.be– Available since 2000– GPL DR supporting Linux (x86, x86_64, powerpc,

sparc and ia64) – Support major Linux distributions– Target backup media includes CDR, DVD-R(w),

tape/OBDR, USB/disks, network (NFS/CIFS)– Understands major file systems including LVM, soft-

and hardware RAID


• mkCDrec– Pro:

• Open Source with excellent support base• Only CLI with kiss in mind• Cloning support• Uses the active Linux kernel on rescue image• Integrates with 3th party backup sw (DP, TSM,…)

– Contra:• Only 1 main developer• Documentation is minimal, but still enough/clear


• Relax and Recover (rear)– rear.sourceforge.net– Available since 2006– GPL DR supporting Linux (x86, x86_64, ia64) – Support major Linux distributions– Target backup media includes CDR, DVD-

R(w), disk, network (NFS/CIFS), rsync– Understands major file systems including

LVM, soft- and hardware RAID


• Relax and Recover (rear)– Pro:

• Open Source with excellent support base• Only CLI with kiss in mind (LSB compliant)• Extremely modular code• Uses the active Linux kernel on rescue image• Integrates with 3th party backup sw (TSM, DP)

– Contra:• Linux kernel => 2.6.x• Slow development (based on sponsoring)• Lacks good documentation

LSB rules of ReaR• Follows the Linux Standard Base rules• Configuration files are under /etc/rear/• The scripts are stored under /usr/share/rear/• One main script /usr/sbin/rear• rear is build around concepts:

– mkrescue– mkbackup– mkbackuponly– recover– dump

Architecture of ReaRrear dump:Dumping out configuration and system informationSystem definition: ARCH = Linux-i386 OS = GNU/Linux OS_VENDOR = FedoraCore OS_VENDOR_ARCH = FedoraCore/i386 OS_VENDOR_VERSION = FedoraCore/6Configuration tree: Linux-i386.conf : OK GNU/Linux.conf : OK FedoraCore.conf : missing/empty FedoraCore/i386.conf : missing/empty FedoraCore/6.conf : missing/empty site.conf : OK local.conf : OK

/etc/rear

FedoreCore GNU

Architecture of ReaR (cont'd)• Shell scripts are stored under /usr/share/rear• Scripts are kept together according workflows

– mkrescue (only make rescue image)– mkbackup (including make rescue image)– mkbackuponly (excluding make rescue image)– recover (the actual recovery part)

• /etc/rear/recovery is being build dynamically

Workflow backup (or rescue)• mkbackup – mkrescue

– Preparation (building the root file system layout)– Analyse (disaster recovery environment creation)

• Creation of /etc/rear/recovery structure

– Analyse (building the rescue system)– Build (copy all executables that are needed)– Pack (kernel and initial ramdisk)– Backup (optional)– Output (copy to destination, PXE, ISO,...)– Cleanup

Workflow recovery• The same configuration files are read

during the recovery workflow• Recovery Process:

– Verify (integrity and sanity check)– Recreate (file system layout)– Restore (the backups including Operating

System)– Finalize (install boot loader, dump recovery

log into /tmp of the recovered system)

Testing on Debian 4 (IA64)

• We tested mondorescue, mkcdrec and rear on the same fresh installed Linux

• IA64 is an new architecture (Linux based)• X86 and x86_64 are well tested and will

most likely work very well for all of them• On the next slides we give our experience• Only used the sources of the 3 products to

start with…

Test mondorescue

• Prerequisites:– gcc, g++, make, gawk, afio, bzip2, cdrecord,

mkisofs, parted, libnewt-dev, lvm2, buffer• Downloaded from ftp.mondorescue.org/src/

– mondo-doc-2.24.tar.gz– mondo-2.2.5.tar.gz– mindi-busybox-1.7.3.tar.gz– mindi-2.0.0.tar.gz

• Make DR archive: mondoarchive

mondoarchive

Boot mondorescue CD

Test mkCDrec

• Prerequisites:– gcc, make, gawk, bzip2, cdrecord, mkisofs,

parted, lvm2, rsync, iproute, bash• Downloaded from mkcdrec.ota.be

– mkCDrec_v0.9.8.tar.gz• Make DR archive:

– Edit Config.sh (change BOOTARCH, ELILO_DIR)– make

make (of mkcdrec)

New features in v0.9.9

Test ReaR

• Prerequisites:– gawk, bzip2, cdrecord, mkisofs, parted, lvm2,

rsync, iproute, mingetty, alien, lsb, bash• Downloaded from rear.sf.net

– rear-1.6.tar.gz and rear-1.6-1.noarch.rpm– Install as follow:

• alien –I rear-1.6-1.noarch.rpm• Edit /etc/rear/Linux-ia64.conf

change agetty into getty

• Make DR archive: rear mkbackup

www.it3.be

Check us out at:

http://www.it3.be/

http://www.it3.be/

linux disaster recovery solutions

Documents