department of computer sciences, university of wisconsin madison dada dynamic allocation of disk...
DESCRIPTION
Department of Computer Sciences, University of Wisconsin Madison Problem Disk Volumes Allocate physical space on creation Multiple Disk Volumes on a physical disk Unused space Disk space not a premium What If?TRANSCRIPT
Department of Computer Sciences, University of Wisconsin Madison
DADA – Dynamic Allocation of Disk Area
Jayaram BobbaVivek Shrivastava
Department of Computer Sciences, University of Wisconsin Madison
OutlineIntroduction Existing Framework Implementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Problem
Disk Volumes Allocate physical space on creation
Multiple Disk Volumes on a physical disk Unused space Disk space not a premium What If?
Department of Computer Sciences, University of Wisconsin Madison
Motivation Virtual Machine Environment
Add Some points
Storage Area Networks Storage area is the bo
Dynamic Physical Allocation (LFS style)
Department of Computer Sciences, University of Wisconsin Madison
Related Work HP AutoRAID
Dynamically change redundancy levels Hot Data – Mirrored Cold Data – RAID 5 Dynamic migration of data
Department of Computer Sciences, University of Wisconsin Madison
Outline IntroductionExisting Framework Implementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Existing Framework Logical Volume Manager (LVM)
User space tool that enables creation of logical volumes of a logical partition.
Supports resizing of volumes. Device Mapper (DM)
Kernel driver that provides a level of indirection in address translation
Department of Computer Sciences, University of Wisconsin Madison
LVM design Volume Group (VG) Physical Volume (PV) Logical Volume (LV) Physical Extent (PE) Logical Extent (LE)
Department of Computer Sciences, University of Wisconsin Madison
LVM design
Physical Device
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LV 1
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM design
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
LVM designVG
PV
LE
LV 1LV2
PE
Department of Computer Sciences, University of Wisconsin Madison
DM designUserspace Applications
log kcopyd
emc
ioctl interface filesystem interface
control interface block interface
core device-mappermapping/target interface
linear striped snapshot multipath
round-robin
path selectors
hardwarehandlers
Department of Computer Sciences, University of Wisconsin Madison
LVM – DM interaction
Kernel
Userspace
Device Mapper
libdevmapper
LVM2
Department of Computer Sciences, University of Wisconsin Madison
Outline Introduction Existing FrameworkImplementation Issues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Modifications A trap mechanism from kernel driver to user-
level LVM code.
Support for multiple segments in LV
Support for virtual mappings in driver
Department of Computer Sciences, University of Wisconsin Madison
Initial State
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
PE (4MB) 25 Extents
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon Create LV
32MB
User A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Map 0-7 extents for user A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Map 0-7 extents for user A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
map 8-11 extents for B
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
map 8-11 extents for B
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User BRead/write
Read/write
Department of Computer Sciences, University of Wisconsin Madison
Volume Group Creation
LVM
Device Mapper
User space
Kernel space
Unallocated Disk Space
LVM daemon
PE (4MB) 25 Extents
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon Create LV1
32MB
User A
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon
User A
Virtually map 0-7 extents for user A
{0-7 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV
16 MB
{0-7 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
Create LV2
16 MB
Virtually map 8-11 extents for B
{0-7 error} {8-11 error}
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
Write 6MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
On Demand allocation of disk
area
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}
{8-11 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}{8-11 error}
Write
10MB
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
(0-1 linear}{0-7 error}{8-11 error}
Write 10 MB trap
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space
Kernel space
Free Disk Space
LVM daemon User A
User B
{0-7 error} {8-11 error}
writetrap
On Demand allocation of disk
area
Department of Computer Sciences, University of Wisconsin Madison
Current Implementation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
User B
{0-1 linear}{2-7 error}{8-10 linear}
Write 10 MBtrap
On Demand allocation of disk
area
Allocate 3 extents (12 MB)
{11 error}
Department of Computer Sciences, University of Wisconsin Madison
Outline Introduction Existing Framework ImplementationIssues Results Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Issues Not the most efficient implementation On a trap,
How much to allocate? On demand Pre-allocate
Where to allocate? Temporal locality Spatial locality
Free unused space
Department of Computer Sciences, University of Wisconsin Madison
How much? On-demand
Allocate only as many blocks as needed Pre-allocate
Pre-allocate physical space for blocks likelyto be accessed in the future.
How much? Where?
Department of Computer Sciences, University of Wisconsin Madison
How much? Tradeoff – Performance vs. Disk Space Depends on the workload
I/O on critical path Asynchronous I/O Multithreaded Applications
Department of Computer Sciences, University of Wisconsin Madison
Pre-Allocation
Think: Filesystem Block Allocation Dumb Allocation
Stride Allocation
Multi-Strided Allocation
Department of Computer Sciences, University of Wisconsin Madison
Dumb Allocation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
{0-1 linear}{2-7 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 2 extents (8MB)
Department of Computer Sciences, University of Wisconsin Madison
Stride Allocation
LVM
Device Mapper
User space Kernel space
Free Disk Space
LVM daemon User A
{0-1 linear}{2-7 error}
Write 6MB
trap
On Demand allocation of disk
area
Allocate 4 extents (16MB)
PreAllocated Extent
Department of Computer Sciences, University of Wisconsin Madison
Ext2 access pattern
Department of Computer Sciences, University of Wisconsin Madison
Ext3 access pattern
Department of Computer Sciences, University of Wisconsin Madison
Outline Introduction Existing Framework Implementation IssuesResults Conclusions
Department of Computer Sciences, University of Wisconsin Madison
Results Micro Benchmarks
mkfs Trace based analysis
Trace collected by HP Research Lab
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : mkfs
ExecutionTimes
0
20
40
60
80
100
120
140
8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
Extent Size
Tim
e Ta
ken(
in s
ecs)
#Traps Taken
0
50
100
150
200
250
300
350
400
450
8K 16K 32K 64K 128K 256K 512K 1M 2M 4M
Extent Size
# Tr
aps
Take
n
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : Asynchronous I/O
Department of Computer Sciences, University of Wisconsin Madison
Micro Benchmark : Asynchronous I/O
Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Department of Computer Sciences, University of Wisconsin Madison
Trace Based Analysis
Department of Computer Sciences, University of Wisconsin Madison
Outline Introduction Existing Framework Implementation Issues ResultsConclusions
Department of Computer Sciences, University of Wisconsin Madison
Conclusions Tradeoff between performance penalty and
space Useful in environments where space is the
bottleneck resource Techniques like Pre Allocation reduce the
performance penalty to some extent For practical implementation, LVM must be
placed in kernel address space
Department of Computer Sciences, University of Wisconsin Madison
Conclusions mkfs performance degrades substantially due
to traps. No degradation in performance for CPU
intensive applications with asynchronous I/O Extent size in the range of 512K-1M provide
best results for the trace This evaluation will be more realistic, if LVM
is moved into Kernel Address space.