backups

using rsync and duplicity

Who needs to backup?

There are 2 kinds of people in this world:1. Those who keep backups.2. Those who lose their data.

What is a backup?

The copying and archiving of computer data to assist in recovering from a data loss event.

What causes data loss?

● Failure● Disaster● Crime● Intentional action● Unintentional action

Why keep a backup?

● Keeps important files safe● Helps us recover from stupid mistakes● Helps recover from malicious attacks● CYA ( Cover Your Ass )

Types of backups

● RAID!?

RAID?!

RAID is not a backup!

ಠ_ಠ

Types of backups

● RAID!?● Unstructured

Types of backups

● RAID!?● Unstructured● Differential● Incremental

Rsync

● extremely efficient file transfer protocol for Unix● open source (GNU license)● written by Andrew Tridgell and Paul Mackerras at the

Australian National University in 1996

Rsync

We can backup to either local or remote server

LOCALGo to external disk for local backup. Backing up to same hard drive is not sufficient for a comprehensive backup solution!

EXTERNALGoes to rsync daemon (can be standalone or started from rsync client by inetd). Route through SSH shell for security.

Rsync

Local locations:~/docs/important

Remote locations:Remote Shell[user@]server.com:~/docs/important

Rsync Daemon[user@]server.com::~/docs/importantrsync://[user@]server.com/~/docs/important

Rsync

common flags:● -v : verbose

Rsync

common flags:● -v : verbose

● -a : archive

Rsync

common flags:● -v : verbose

● -a : archive

● -z : compress

Rsync

common flags:● -v : verbose

● -a : archive

● -z : compress

● --rsh=/usr/bin/ssh

Rsync

common flags:● -v : verbose

● -a : archive

● -z : compress

● --rsh=/usr/bin/ssh

● --delete : delete extraneous files

Rsync

common flags:● -v : verbose

● -a : archive

● -z : compress

● --rsh=/usr/bin/ssh

● --delete : delete extraneous files

● --link-dest : create hard links (incremental backup)

Incremental backups with Rsync

Hard links allow us to create extra nodes for our files so we can make sure at least one copy exists without using extra space.

Advantages of Rsync over cp

● Mirroring data● Transmitting lots of data● Only copy what is needed● Use with --link-dest to create hard links for

incremental backups

Duplicity

● First developed in 2002.● Current version is 0.6.19 released in May

2012● GNU licensed● Written in Python & C● Leverages Rsync algorithm● Available via package managers, or via

RPM, binaries, or source downloadable from http://duplicity.nongnu.org/

Duplicity

Produces tar-format volumes on local or remote servers

Duplicity files

● Backup● Manifest● Signature

Duplicity filesystems

Local File Storage, scp/SSH, Rsync, FTP, HSI, WebDAV, Tahoe-LAFS, Amazon S3, with others easily added

Duplicity

Remote locations are specified using URIs:● ftp://[user@]archive.business.com/backups● rsync://[user@]newspeak.marketing.biz/outside_box/● s3+http://awesometown-socool

Using Duplicity

By default, Duplicity doesn't need any command line options to make encrypted, incremental backups to a local or remote location. However, for a sane backup strategy we should:

● Perform periodic full (snapshot) backups● Delete old backups● Ignore certain files

Using Duplicity

● --full-if-older-than

Using Duplicity

● --full-if-older-than● remove-all-but-n-full

Using Duplicity

● --full-if-older-than● remove-all-but-n-full● --include/exclude-globbing-filelist

Using Duplicity

● --full-if-older-than● remove-all-but-n-full● --include/exclude-globbing-filelist● PASSPHRASE environment variable

Moar Duplicity

Also used to verify backups and restore backups

Verify:duplicity verify [backup] [source]can also be used to determine changes since backup

Moar Duplicity

Also used to verify backups and restore backups

Verify:duplicity verify [backup] [source]can also be used to determine changes since backup

Restore:duplicity restore [backup] [destination]Can use --file-to-restore and --time flags to specify more precise restoration

Why use Duplicity over Rsync?

● Encrypted and signed archives● Built in incremental backup● Ability to use a wide range of local and

remote filesystems

Best Practices

● Trust, but verify● Use strong passphrase● Backup to multiple locations● Automate

Automation

For the best backup strategy, we take the human out of the equation.

● write a Bash script and cron it● Use a GUI

Related LinksJamie Zawinski on backupshttp://www.jwz.org/doc/backups.htmlSnapshot Backups with Rsync by Mike Ruebelhttp://www.mikerubel.org/computers/rsync_snapshots/Epydocs for Duplicityhttp://duplicity.nongnu.org/epydoc/index.htmlDuplicity to Amazon S3 Bash Scripthttp://blog.damontimm.com/bash-script-incremental-encrypted-backups-duplicity-amazon-s3/RTFM!http://linux.die.net/man/1/rsynchttp://duplicity.nongnu.org/duplicity.1.htmlLots of Horror Stories about not Backing Uphttp://reddit.com/r/talesfromtechsupport