Exo-clonesInstantaneous Change Tracking for the Whole File SystemRichard P. Spillane, Wenguang Wang, Luke Lu, Maxime Austruy, Christos Karmanolis, Rawlinson Rivera

May 10th, 2016

VDFS (POSIX)

VSAN

VDFS (POSIX)

DeveloperLaptop

Backups, Documents,Applications

Exo Support Exo Support

1Hot Storage 2016

rsync/diff -R

Exo-clone

What is an Exo-clone?

2

A

FS SnapshotUpdatedSnapshot

A’

File Data

Meta Data A

FS SnapshotUpdatedSnapshot

A’

File Data

Meta Data

VDFS on VSAN VDFS on Developer Laptop

Copy of A’

Scan whole FS

• “free” to store• “free” to create• “free” to import

skipped

Hot Storage 2016

Benefits of Exo-clone

3

No duplication

No FS Scanning

diff -R ✖ ✖

git pack-objects ✔ ✖

Docker FS layers ✖ ✔

ZFS/BtrFS send/receive ✖ ✔

Exo-clones ✔ ✔

Hot Storage 2016

Exo-clone Use Cases

4

Changes in last hour

Application changes

Backup/replication

Application Revision Control

Improve with Exo-clones?

Hot Storage 2016

A Common Problem…

5

MyApp 1.6

Sources…git

Images, 3D models,icons, sounds, …?100sMB—10sGB

OS & Libraries…Docker

Lots of changes to large andsmall files

Backup MyApp

Hot Storage 2016

git? Sadly, no…

6

WORKING OBJECT STORE

A B

Redundant copy, 5-10x bigger thancurrent working state

checkout/commitcopy GBs of data

Hot Storage 2016

Docker?

7

Ubuntu14.04.2

MyApp1.0

network

Docker pullDocker push

Dockerfile:FROM 14.04.2apt-get install MyLibcp build/ /opt/mycp assets/ /opt/assets

Hot Storage 2016

Docker FS Layer

… = FS Layer 0x7dfb…

… = FS Layer 0xe9a0…

Other filesare the same

8Hot Storage 2016

Update With Docker Pull

9

Ubuntu14.04.3

MyApp1.1

network

Ubuntu14.04.3

Ubuntu14.04.2

Ubuntu14.04.2

MyApp1.0

200MB

MyApp1.1

MyApp1.0

…downstream

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.3

MyApp1.1

Ubuntu14.04.3

MyApp1.1

Security update

50MB250MB250MB

Hot Storage 2016

Update With Docker Pull

10

Ubuntu14.04.3

MyApp1.1

network

Ubuntu14.04.3

Ubuntu14.04.2

Ubuntu14.04.2

MyApp1.0

200MB

MyApp1.1

MyApp1.0

…downstream

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.3

MyApp1.1

Ubuntu14.04.3

MyApp1.1

Security update

50MB250MB250MBdocker build, push, pull, Dockerfile, …

Change Tracking FS

AUFS and LayersExo-clones

Hot Storage 2016

Updating Base Layers With Exo-clones

11

Ubuntu14.04.3

network

Docker pullDocker push

E-cUbuntu14.04.2

.3 - .2

network

Security update

Docker workflow Exo-clones

Hot Storage 2016

Update With Docker Utilizing Exo-clones

12

Ubuntu14.04.3

MyApp1.1

network

Ubuntu14.04.3

Ubuntu14.04.2

Ubuntu14.04.2

MyApp1.0

200MB

MyApp1.1

MyApp1.0

…downstream

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.2

MyApp1.0

Ubuntu14.04.3

MyApp1.1

Ubuntu14.04.3

MyApp1.1

Security update

50MB250MB250MB

.3 - .2

25MB

.3 - .2.3 - .2

75MB75MB

Still sending all of MyApp

.1 wrt .0

Dedup wrt to non-parental

volumes

5MB30MB30MB

Hot Storage 2016

Network Overhead of Docker FS Layers vs. Exo-clonesTransfer size of Docker FS layers compared to Exo-clones using whole-file dedup and Exo-clones using either block or 6-9KB variable length chunking

13

0

50

100

150

200

250

300

350

400

LTS Upgrade PXE Upgrade (zipped)

Docker FS Layer Exo Whole-file Exo Blk-chunk Exo Var-chunk

Laye

r Siz

e (M

B)

88%

25%

Hot Storage 2016

Exo-clone Format

14

1. rm /a/12. curl http://img.com/huge > /new/foo3. commit /.exo/clone1

clone1:

rm /a/1

touch /new/foo

map /new/foo to yellow data block

Ref Count=2Ref Count=1Ref Count=2

Meta-dataSection

DataSection

/new/foo/a/1 /a/2 /.exo/clone1

Ref Count=2Ref Count=1

Ref Count=1

Parent SS…

Child SS:

Only additional space used by

Exo-clone

Hot Storage 2016

Summary• Instantly create and share diffs between any two FSes with

– No space overhead– No sending the same block twice over the wire

• Better backend for Docker, rsync, git• Easy but efficient backup/replication between any storage environment• What becomes simpler when we make revision control on the whole FS cheap and standard?

– What if every FS (e.g., ZFS, BtrFS, LVM, NASes) supported exo-clones?

15Hot Storage 2016

Thank Yourspillane@vmware.comhttp://telegram.me/necro351

16Hot Storage 2016

Zero-copy Requires Block Alignment

17

A

SnapshotUpdatedSnapshot

A’

BlockStore

Meta Data

Existing FS

send/receive

A

SnapshotUpdatedSnapshot

A’

BlockStore

Meta Data

Zero-copyExo-clone

Without Zero-copy With Zero-copy

Small files/changes increase exo-

clone size

Zero-copy export (and

import)

Hot Storage 2016

How do exo-clones compare to ZFS and BtrFS send/receive?

0

100

200

300

400

500

600

700

800

Linux Django Ubuntu

tar.gztarBtrFSZFSVDFS

Volumes full of small files are encoded less efficiently in exchange for zero-copy

We rely on an additional gzip pass when transferring over network

18

Laye

r Siz

e (M

B)

Hot Storage 2016

Exo-Clone Prototype doing Cassandra Migration

WAN

VSAN

VDFS

Deployment Cluster

VDFS

VSAN

VDFS

Test/Dev or Backup Cluster

VDFS

19Hot Storage 2016

Application Revision Control

20

MyApp 1.1

MyApp 1.6 mobile

MyApp 1.4

MyApp 1.6, 1.5, …

MyApp 1.7a

MyApp 1.7b

MyApp 1.6

Bugs & Feature RequestsFeatures & Perf & Fixes & Bugs

Hot Storage 2016

VDFSExo Support

VDFSExo Support

Implementation of Exo-clones on VDFS

21

VSAN AWS/EBS (?)POSIX (e.g., Ext4)

Exo-clon

e VDFSExo Support

Hot Storage 2016

git? Sadly, no…

22

WORKING OBJECT STORE

A B

1.Checkoutcopies GBs

2.Modify3.Add

4.Commitcopies GBs

git add…

Redundant copy, 5-10x bigger thancurrent working state

Hot Storage 2016