life preserver - asiabsdcon2014

AsiaBSDCon 2014AsiaBSDCon 2014AsiaBSDCon 2014AsiaBSDCon 2014

Snapshots, Replication, and Boot-EnvironmentsSnapshots, Replication, and Boot-Environments--

How new ZFS utilities are changing FreeBSD & PC-BSD. How new ZFS utilities are changing FreeBSD & PC-BSD.

Kris MooreKris MoorePC-BSD / iXsystemsPC-BSD / iXsystems

[email protected]@pcbsd.org

Why ZFS?Why ZFS?

Why ZFS?Why ZFS?

● The benefits have begun to greatly The benefits have begun to greatly outweigh the drawbacks for desktop outweigh the drawbacks for desktop usageusage

● Since moving to only 64bit for 9.2 and Since moving to only 64bit for 9.2 and later, the last major hurdle has been later, the last major hurdle has been overcomeovercome

● A new class of utilities and methods have A new class of utilities and methods have been made possible by building on top of been made possible by building on top of ZFSZFS

A closer look at Boot-EnvironmentsA closer look at Boot-Environments

A closer look at Boot-EnvironmentsA closer look at Boot-Environments

● First originated in Solaris 10First originated in Solaris 10● Provides a method of using ZFS Provides a method of using ZFS

snapshots and clones to create instant snapshots and clones to create instant bootable backups of a systembootable backups of a system

● Most commonly used before doing Most commonly used before doing “dangerous” tasks, such as updating “dangerous” tasks, such as updating kernels, world or critical packageskernels, world or critical packages

PC-BSD boot-environment implementationPC-BSD boot-environment implementation

● GRUB used to perform the direct bootup GRUB used to perform the direct bootup of FreeBSD kernel (using kfreebsd)of FreeBSD kernel (using kfreebsd)

● Special ZFS dataset layoutSpecial ZFS dataset layout● ““beadm” command and GUI utility for beadm” command and GUI utility for

managementmanagement

So why GRUB?So why GRUB?

● Ability to easily tie into “beadm”, provide Ability to easily tie into “beadm”, provide menus and sub-menus for Boot-menus and sub-menus for Boot-EnvironmentsEnvironments

● It's what Solaris usesIt's what Solaris uses● In 2014 it is nice to have a graphical boot-In 2014 it is nice to have a graphical boot-

loaderloader● Fast loading of Kernel + ModulesFast loading of Kernel + Modules

Can I still use the BSD loader?Can I still use the BSD loader?

● Yes, in 10.0 the BSD loader can be Yes, in 10.0 the BSD loader can be selected during installationselected during installation

● It does It does notnot have automatic integration with have automatic integration with “beadm”“beadm”

Will PC-BSD switch back to the BSD loader?

Will PC-BSD switch back to the BSD loader?

● Eventually, as the BSD loader begins to Eventually, as the BSD loader begins to gain the features that we currently use in gain the features that we currently use in GRUBGRUB

A closer look at the ZFS layoutA closer look at the ZFS layout

● By default a <pool>/ROOT/default dataset By default a <pool>/ROOT/default dataset is created, which is mounted on “/”is created, which is mounted on “/”

● This dataset will be snapped / cloned to This dataset will be snapped / cloned to create new BEscreate new BEs


● Any additional ZFS datasets are Any additional ZFS datasets are notnot included in the BEincluded in the BE

● We create /usr and /var datasets with the We create /usr and /var datasets with the “canmount=off” flag“canmount=off” flag

So how do I manage these Boot-Environments?

So how do I manage these Boot-Environments?

● From the command-line, using the From the command-line, using the “beadm” utility“beadm” utility

● When creating / removing new BE's, When creating / removing new BE's, GRUB configuration will be automatically GRUB configuration will be automatically updated with new boot entriesupdated with new boot entries

Via the CLIVia the CLI

Via the GUIVia the GUI

● Can be accessed via Control Panel → Can be accessed via Control Panel → Boot ManagerBoot Manager

So how do I boot it?So how do I boot it?

● When only a single BE exists, the menu is When only a single BE exists, the menu is hidden unless Left-Shift is held downhidden unless Left-Shift is held down

● Once BE's are created, the menu will Once BE's are created, the menu will appear, booting the first entry by defaultappear, booting the first entry by default

Customizing the GRUB menusCustomizing the GRUB menus

● Most GRUB customization can be done Most GRUB customization can be done via knobs in /usr/local/etc/default/grubvia knobs in /usr/local/etc/default/grub

● After making changes to this file, run After making changes to this file, run “grub-mkconfig -o /boot/grub/grub.cfg”“grub-mkconfig -o /boot/grub/grub.cfg”

● /boot/grub/grub.cfg contains the bootup /boot/grub/grub.cfg contains the bootup script, written in mostly 'sh' syntaxscript, written in mostly 'sh' syntax

Common CustomizationsCommon Customizations

● GRUB_HIDDEN_TIMEOUT_QUIET=true/falseGRUB_HIDDEN_TIMEOUT_QUIET=true/false● GRUB_HIDDEN_TIMEOUT=<number>GRUB_HIDDEN_TIMEOUT=<number>● GRUB_DEFAULT=<number>GRUB_DEFAULT=<number>● GRUB_TIMEOUT=<number>GRUB_TIMEOUT=<number>● GRUB_THEME=<theme path>GRUB_THEME=<theme path>● GRUB_FONT=<pf2 file>GRUB_FONT=<pf2 file>

A closer look at Life-PreserverA closer look at Life-Preserver

What is this Life-Preserver utility?What is this Life-Preserver utility?

● A command-line and graphical front-end A command-line and graphical front-end to:to:

● Scheduled ZFS snapshots / pruningScheduled ZFS snapshots / pruning● Scheduled ZFS send/recv replicationScheduled ZFS send/recv replication● Zpool monitoringZpool monitoring● Bare-Metal restore using PC-BSD mediaBare-Metal restore using PC-BSD media

How do I get this utility?How do I get this utility?

● Included out of box on PC-BSD 10.x and Included out of box on PC-BSD 10.x and laterlater

● Via FreeBSD ports / packages:Via FreeBSD ports / packages:● sysutils/pcbsd-utils (Command-line)sysutils/pcbsd-utils (Command-line)● sysutils/pcbsd-utils-qt4 (GUI front-ends)sysutils/pcbsd-utils-qt4 (GUI front-ends)

Scheduling SnapshotsScheduling Snapshots

● To get started with a snapshot schedule, To get started with a snapshot schedule, we can use the following command:we can use the following command:

# lpreserver cronsnap tank1 start daily@22 10# lpreserver cronsnap tank1 start daily@22 10

● Replace “tank1” with your zpoolReplace “tank1” with your zpool

● daily@XX / hourly / 30min / 10min / 5mindaily@XX / hourly / 30min / 10min / 5min

● Replace 10 with number to keepReplace 10 with number to keep

What does that command do?What does that command do?

● Creates the following entry in /etc/crontabCreates the following entry in /etc/crontab

0 14 * * * root /usr/local/share/lpreserver/backend/runsnap.sh tank1 100 14 * * * root /usr/local/share/lpreserver/backend/runsnap.sh tank1 10

● When run, the script does the followingWhen run, the script does the following

● Confirm zpool / datasetConfirm zpool / dataset● Create new snapshot, recursivelyCreate new snapshot, recursively● Selectively auto-prune snapshotsSelectively auto-prune snapshots● Send out notification e-mail (if enabled)Send out notification e-mail (if enabled)● Start auto-replication (if enabled)Start auto-replication (if enabled)


Recent UpdateRecent Update

● New “auto” mode for snapshot schedulingNew “auto” mode for snapshot scheduling● Added for 10.0.1 / quarterly PC-BSD Added for 10.0.1 / quarterly PC-BSD

updateupdate


● Snapshots will be created every 5 minutes Snapshots will be created every 5 minutes and kept for an hour.and kept for an hour.

● A hourly snapshot will be kept for a day.A hourly snapshot will be kept for a day.● A daily snapshot will be kept for a month.A daily snapshot will be kept for a month.● A monthly snapshot will be kept for a year.A monthly snapshot will be kept for a year.


● The life-preserver daemon will also keep The life-preserver daemon will also keep track of the zpool disk spacetrack of the zpool disk space

● If the capacity falls below 75%, the oldest If the capacity falls below 75%, the oldest snapshot will be auto-pruned.snapshot will be auto-pruned.

How do I enable e-mail notifications?How do I enable e-mail notifications?

● Using the command:Using the command: # lpreserver set <email> (Uses the “mail” command)# lpreserver set <email> (Uses the “mail” command)

● Additional commands:Additional commands:● # lpreserver set duwarn <percent># lpreserver set duwarn <percent>

● Will send warning when disk free falls below %Will send warning when disk free falls below %

● Additional commands (continued):Additional commands (continued):● # lpreserver set emailopts ALL/WARN/ERROR# lpreserver set emailopts ALL/WARN/ERROR

● Send mail for all actions, warnings + errors, or errors onlySend mail for all actions, warnings + errors, or errors only

What about replication?What about replication?

● Once snapshots are enabled, replication can be Once snapshots are enabled, replication can be setup to run automatically, or at a specific setup to run automatically, or at a specific intervalinterval

● Replication by default runs over SSHReplication by default runs over SSH● Uses ZFS send/recv (Requires that target have Uses ZFS send/recv (Requires that target have

supported ZFS version)supported ZFS version)

Starting replicationStarting replication

● First, prepare a remote systemFirst, prepare a remote system● Create a ZFS dataset / SSH userCreate a ZFS dataset / SSH user● Now set ZFS allow properties on datasetNow set ZFS allow properties on dataset

# zfs allow -u <user> create,receive,mount,userprop,destroy,send,hold <dataset># zfs allow -u <user> create,receive,mount,userprop,destroy,send,hold <dataset>

● Next start replication on systemNext start replication on system

# lpreserver replicate add <host> <user> 22 tank1 remotepool/backups sync# lpreserver replicate add <host> <user> 22 tank1 remotepool/backups sync

● Replace <host>, <user>, 22/Port, etcReplace <host>, <user>, 22/Port, etc

What does this replication do?What does this replication do?

● Checks specified zpoolChecks specified zpool● Checks “backup:lpreserver” ZFS property for Checks “backup:lpreserver” ZFS property for

last replicationlast replication● Check if doing a first-time / “full” replication or Check if doing a first-time / “full” replication or

else perform incrementalelse perform incremental● Start the ZFS send/recv commandsStart the ZFS send/recv commands● After successful send, mark “backup:lpreserver” After successful send, mark “backup:lpreserver”

property with last snapshot replicatedproperty with last snapshot replicated

What does this replication do? (Contd)What does this replication do? (Contd)

● Build a complete list of zpool / dataset Build a complete list of zpool / dataset propertiesproperties

● Save this list to remote systemSave this list to remote system● Check if e-mail reporting is enabled, send Check if e-mail reporting is enabled, send

results / logsresults / logs

Notes on replicationNotes on replication

● First time replication can take some timeFirst time replication can take some time● During replication new snapshots can be During replication new snapshots can be

created, but auto-pruning is disabledcreated, but auto-pruning is disabled● If a replication fails, it may be required to re-If a replication fails, it may be required to re-

initialize the remote side.initialize the remote side.● # lpreserver replicate init <mypool># lpreserver replicate init <mypool>

Getting my stuff back!Getting my stuff back!

● Once snapshots are being created, there are Once snapshots are being created, there are several ways to revert or restore filesseveral ways to revert or restore files

● Via the CLI you can run:Via the CLI you can run:

# lpreserver revertsnap <dataset> <snap># lpreserver revertsnap <dataset> <snap>

● Additionally you can browse files in the Additionally you can browse files in the <mountpoint>/.zfs/snapshot directory<mountpoint>/.zfs/snapshot directory


● Via the GUI utility, it is possible to Via the GUI utility, it is possible to “browse” snapshot data, as well as scroll “browse” snapshot data, as well as scroll backwards in time.backwards in time.

How about bare-metal restores?How about bare-metal restores?

● The PC-BSD 10.x install DVD/USB media can The PC-BSD 10.x install DVD/USB media can provide “bare-metal” restore functionalityprovide “bare-metal” restore functionality

● Currently limited to the GUI install, but will be Currently limited to the GUI install, but will be added to the text-installer in the futureadded to the text-installer in the future

● Allows you to adjust / change zpool optionsAllows you to adjust / change zpool options

What does a bare-metal restore look like?What does a bare-metal restore look like?

The bare-metal nitty grittyThe bare-metal nitty gritty

● PC-BSD uses the pc-sysinstall installation PC-BSD uses the pc-sysinstall installation backendbackend

● Supports the “zfsrestore” installMode, Supports the “zfsrestore” installMode, allowing installs from a ZFS replicationallowing installs from a ZFS replication

● Uses SSH as the transfer agentUses SSH as the transfer agent● Uses normal pc-sysinstall disk-setupUses normal pc-sysinstall disk-setup

The bare-metal nitty gritty (contd)The bare-metal nitty gritty (contd)

● Example pc-sysinstall config Example pc-sysinstall config installMode=zfsrestore installMode=zfsrestore

sshHost=backuphostsshHost=backuphost

sshUser=backupusersshUser=backupuser

sshPort=22sshPort=22

sshKey=/root/.ssh/backupkey_rsasshKey=/root/.ssh/backupkey_rsa

zfsProps=.lp-props-backups-systemnamezfsProps=.lp-props-backups-systemname

zfsRemoteDataset=/backups/systemnamezfsRemoteDataset=/backups/systemname

What about ZFS mirroring?What about ZFS mirroring?

● During system installation, you may have setup During system installation, you may have setup with a single-disk, or raidz.with a single-disk, or raidz.

● Life-Preserver can “attach” new disks, creating Life-Preserver can “attach” new disks, creating a mirror or adding additional raidz disks.a mirror or adding additional raidz disks.

What about ZFS mirroring? (Contd)What about ZFS mirroring? (Contd)

● Attaching a new disk to existing zpoolAttaching a new disk to existing zpool

# lpreserver zpool attach tank1 /dev/da0# lpreserver zpool attach tank1 /dev/da0

● Life-Preserver takes the following stepsLife-Preserver takes the following steps● Wipes partitions on new diskWipes partitions on new disk● Creates matching GPT/MBR partitionCreates matching GPT/MBR partition● Makes the disk bootable with GRUBMakes the disk bootable with GRUB● Inserts disk into zpool to begin resilver Inserts disk into zpool to begin resilver

processprocess

Questions?Questions?

ReferencesReferences

BSDCan pc-sysinstall talk 2010BSDCan pc-sysinstall talk 2010http://www.bsdcan.org/2010/schedule/events/173.en.htmlhttp://www.bsdcan.org/2010/schedule/events/173.en.html

life preserver - asiabsdcon2014

Technology

bsd loader

bootenvironmentsa closer

zfs layouta closer look

tru grub

grub menuscustomizing

new zfs utilities

new boot entries

boot environmentsenvironments