unioning of the buffer cache and journaling layers with ... · unioning of the buffer cache and...

Unioning of the Buffer Cache and Journaling Layers with Non-volatile Memory

USENIX FAST „13

Eunji Lee (Ewha University, Seoul, Korea)

Hyokyung Bahn (Ewha University)

Sam H. Noh (Hongik University)

Outline

Reliability issues in storage systems

Non-volatile memory as a solution

UBJ: Unioning of Buffer cache and Journaling

Performance evaluation

2

A man working hard …

3

What the #$%^*&?

A man working hard …

4

What the #$%^*&?

Sudden power failure incurs file system inconsistency

G O E A S T

So what happened?

5

T S A E

Buffer cache

W

Power went off N

E W

Volatile

Non-volatile

After reboot..

S

T S A E W W E “ ”

File system

Journaling as a solution

Prevent data inconsistency through write-twice

ext4, ReiserFS, XFS, btrFS …

6

S G O E A S T

Buffer Cache

T S A E W E W E

W E Journal Area

W E

Commit

Checkpoint

N

E W

Consistency guaranteed!

File system

6.8

0

0.5

1

1.5

2

2.5

3 Journal no-Journal

Increase storage writes by 2.7 times

Non-volatile

Memory

7


DRAM

Journal area

Buffer cache

File system

Non-volatile memory as main memory

8

(Ming-Hsiu Lee Macronix, NVMTS 2011)

1. Scaling Limit of DRAM

“DRAM technology is greatly

challenged beyond 45nm “

(NVMW ‟10, Driskill)

2. Power consumption

As much as 40% of the total system

energy is consumed by the main memory

subsystem in a mid-range IBM eServer

machine. (Querish, ISCA 2009)

3. Demand for fast memory access

Replacing DRAM with STT-RAM in data

centers can reduce power by up to 75%

(NVMW ‟10, Driskill)

As critical applications are becoming

more data-centric, memory

performance is fast becoming the key

bottleneck

Non-volatile Memory Technology

9

Scalability Low-power High-performance

Source: T. Perez, C. A. F. D Rose, Technical Report, PUCRS, 2010

Non-volatile Memory Technology

10

Scalability Low-power High-performance v v

Source: T. Perez, C. A. F. D Rose, Technical Report, PUCRS, 2010

v (Optimistic expectations)

Seems to provide data consistency

G O E A S T

File system

T S A E


11

T S A E

Buffer Cache

W E W E

W

Power crash

E

N

E W

S

After reboot, restore consistency with remaining data

Non-volatile memory as a solution?

12

Inconsistency problem still exists with NVM

G O E A S T

File system

T S A E

T S

Buffer cache

W E

W G

G S

System crashes during overwriting data in cache

E W

“W” is evicted by cache replacement N

N G O E A S T T S E W G S

If using data in main memory,

G O E A S T T S A W G

If using data in storage ,

Inconsistent

After reboot

Inconsistent

S Drop-in replacement of non-volatile memory

does not suffice

File system

Provide data consistency without sacrificing performance

Design a novel buffer cache architecture “UBJ”

Subsume functions of caching and journaling

Use data block for dual purposes

Provide journaling effect through transition of cache block state

Unioning of Buffer cache and Journaling Layers (UBJ)

13

Non-volatile

memory

G O S T

Buffer Cache

W E

Journal Area

T S W E

Transaction

management W E

Workings of UBJ

14

G O E A S

T S A E

Main memory

T

Secondary Storage

(HDD, Flash, etc.)

T S A E

Cache

update

(W)

start

Cache

update

(E)

Checkpoint

start

Final

update

(W)

Final

update

(E)

Checkpoint

end

end

Commit

W E

Commit Transaction

management

Checkpoint

W E

Cache block (normal)

Cache & log block (frozen)

Role of data block Cache

(normal)

Event sequences

Durability & Consistency

“in-place commit” Freeze blocks

to be write-protected

Change blocks to be writable

Successfully committed data are managed in a

transaction

Workings of UBJ

15

Cache

update

(W)

start

Cache

update

(E)

Checkpoint

start

Final

update

(W)

Final

update

(E)

Checkpoint

end

end

Commit

Cache Cache & log

Role of data block Cache

Event sequences

Log blocks

Transaction Management

Protected from replacement

Copy-on-write for write request

Serve read requests as cache blocks

Please refer to our paper for details!

Buffer

Cache

Cache performance of UBJ

16

Buffer Cache

1. Original buffer cache

Journal area

Journal

Area

Journal area Journal area

Buffer Cache+ Journal area

Secondary storage Secondary storage Secondary storage

2. Separate journaling 3. UBJ

NVRAM NVRAM NVRAM

Cache performance of UBJ

17

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 2

Mis

s ra

tio

cache ratio

BF-JmBF-noUBJ

1. Original

buffer cache 3. UBJ

2. Separate

journaling

UBJ provides nearly same cache performance as original buffer cache

Buffer

Cache Buffer Cache

Journal

Area Buffer Cache+ Journal area

NVRAM NVRAM NVRAM

Cache hits on frozen data blocks

18

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1.0

varmail proxy fileserver webserver

Hit c

ount

dis

trib

ution

hit_when_frozen

hit_when_normal

System recovery

19

Cache

update

(W)

Cache

update

(E) Event sequences

Write ()

start

Checkpoint

start

Final

update

(W)

Final

update

(E)

Checkpoint

end

end

Commit

T S A W

Buffer cache

T S A E E

Crash before commit

Crash during commit

Crash after commit

Consistent

Inconsistent

T S E W

Buffer cache

T S A E W

E W

Consistent

E

Consistent

Transaction

T S E W

Buffer cache

T S A E

Consistent

Inconsistent

Inconsistent

Performance Evaluation

Prototype of UBJ on Linux 2.6.38

Intel Core i3-2100 CPU

3.1GHz and 4GB of DDR2-800 memory

Emulate non-volatile memory with DRAM

Compare with ext4 in journal-mode

logs both data and metadata

Three benchmarks

Filebench, IOzone, Postmark

20


Filebench

21

0

200

400

600

800

1000

1200

1400

1600

1800

2000


Execution t

ime (

s) BF-ext4

UBJ

Improve execution time and throughput by 30.7% and 59.8% on average

0

20

40

60

80

100

120


Thro

ughput(

MB

/s) BF-ext4

UBJ


IOzone

22

0

5

10

15

20

25

30

35

40

45

50

100 150 200 250 300

Thro

ughput(

MB

/s)

Fileset size(MB)

BF-ext4

UBJ

0

20

40

60

80

100

120

140

100 150 200 250 300

Thro

ughput(

MB

/s)

Fileset size(MB)

BF-ext4

UBJ

(a) Random write (b) Sequential write

Improve performance by 110% on average, up to by 240%


Postmark

23

(a) Read (b) Write

0

10

20

30

40

50

60

2K 4K 6K 8K 10K

Thro

ughput(

MB

/s)

Transactions

BF-ext4

UBJ

0

10

20

30

40

50

60

70

80

2K 4K 6K 8K 10K

Thro

ughput(

MB

/s)

Transactions

BF-ext4

UBJ

Improve performance by 109% on average


Effectiveness of UBJ on performance as the commit period changes

24

0.0

1.0

2.0

3.0

4.0

5.0

1 2 3 4 5

Late

ncy (

ms)

Commit period (s)

UBJ

BF-ext4

Latency of ext4 becomes smaller as the commit period is longer

Latency of UBJ is not sensitive to the commit period changes

Conclusion

Novel non-volatile memory buffer cache architecture

Subsumes the functions of caching and journaling

Buffer cache blocks Journal logs

In-place Commit

Notion of a frozen state

Performance results

Implemented on Linux 2.6.38

Compared to ext4 in journal mode

Improve I/O performance by 76% and up to 240%

25

26

Eunji Lee https://sites.google.com/site/alicia0729

Sam H. Noh https://next.hongik.ac.kr

Hyokyung Bahn https://home.ewha.ac.kr/~bahn

Thank you

https://sites.google.com/site/alicia0729

https://sites.google.com/site/alicia0729

https://next.hongik.ac.kr/

https://home.ewha.ac.kr/~bahn

unioning of the buffer cache and journaling layers with ... · unioning of the buffer cache and...

Documents