lzrw3 decompressor dual semester project characterization presentation students: peleg rosen tal...

LZRW3 Decompressordual semester project

Characterization Presentation

Students:Peleg RosenTal Czeizler

Advisors:Moshe PorianNetanel Yamin

6.4.2014

Presentation Content• Project Motivation and Goals• Project Requirements• Background – LZ77 algorithm• LZRW3 algorithm• Compressed file format• Proposed Solution• Decompression Example • Project Top Block Diagram• Planned GUI• Project Schedule and Gantt

Project MotivationWhy do we need an hardware data compression?

Why do we need an hardware data compression?-Reduce storage capacity

Project Motivation

Why do we need an hardware data compression?-Reduce storage capacity-Reduce amount of data to be handled

Project Motivation

Why do we need an hardware data compression?-Reduce storage capacity-Reduce amount of data to be handled-Reduce communication costs

Project Motivation

Why do we need an hardware data compression?-Reduce storage capacity-Reduce amount of data to be handled-Reduce communication costs-Speed

Project Motivation

Project Goals

Project Goals• Implementation of LZRW3 data decompression core.

• Implementation of LZRW3 data decompression core.

• Implementation of a verification environment.

Project Goals

Project Requirements

Project RequirementsPart A:

• Core Requirements:– Process data at the speed of 1 Gbps.– Support data blocks with output of 2KB – 32KB.– Relay only on the FPGA’s internal memory.– VHDL Implementation.

Part A:

• Core Requirements:– Process data at the speed of 1 Gbps.– Support data blocks with output of 2KB – 32KB.– Relay only on the FPGA’s internal memory.– VHDL Implementation.

• Full simulation environment (golden model and checkers).

Project Requirements

Part A:

• Core Requirements:– Process data at the speed of 1 Gbps.– Support data blocks with output of 2KB – 32KB.– Relay only on the FPGA’s internal memory.– VHDL Implementation.

• Full simulation environment (golden model and checkers).

Part B:

• Synthesis & implementation of FPGA device (Xilinx Virtex-5).

Project Requirements

Part A:

• Core Requirements:– Process data at the speed of 1 Gbps.– Support data blocks with output of 2KB – 32KB.– Relay only on the FPGA’s internal memory.– VHDL Implementation.

• Full simulation environment (golden model and checkers).

Part B:

• Synthesis & implementation of FPGA device (Xilinx Virtex-5).

• GUI implementation in VisualStudio.

Project Requirements

Background – LZ77 algorithm

Background – LZ77 algorithm Lossless data compression algorithm that was published by

Abraham Lempel and Jacob Ziv in 1977.

Output item : (offset , length ) + next literal In this case Output item = ( , ) Meaning :

BABDAC ABDBCAA

History buffer Look-ahead buffer

3 245 151 2 33B

Background – LZ77 algorithm Lossless data compression algorithm that was published by

Abraham Lempel and Jacob Ziv in 1977.

-> Take 3 literals -> Next literal will be BGo 5 steps back

Output item (Copy item): [slot address , length ] In this case Output item = [ , ]

BABDACABDBCAA21 30 3

LZRW3 compression algorithm

Hash Function

Hash Table

ABD

Slot address

1

Slot address

4 5 6

Slot address

BAB

0

ABD

Slot address

Send every 3 literals to the hash function

Put offset in the hash table

If the slot is occupied and the literals match - make copy item

6

Compressed file format

Structure

StructureFile header (8 byte)

Compressed file format

StructureFile header (8 byte)Groups:

Compressed file format

StructureFile header (8 byte)Groups: - control bytes (2 bytes)

Compressed file format

Compressed file format

StructureFile header (8 byte)Groups: - control bytes (2 bytes) - data bytes (16 - 32 bytes)* The last group might be smaller

Compressed file format

File headercontains information about the file size and whether it is compressed or not.

Compressed file format

Control bytesHold the information about the items type (copy or literal).

Compressed file formatLiteral itemsSingle bytes of data in their original form.

Compressed file format

Copy itemsTwo bytes representing hash table slot holding the offset and length of a literal sequence.

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

File decompressed!

File notcompressed

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

Control bytes decoding Determine number and location of copy items.

File decompressed!

Filecompressed

File notcompressed

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

Control bytes decoding Determine number and location of copy items.

File decompressed!

Filecompressed

File notcompressed Group decoding

Literal items: 1. Store in hash table2. Copy

Copy items:1. Decode2. Replace3. Copy

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

Control bytes decoding Determine number and location of copy items.

File decompressed!

Filecompressed

File notcompressed

Next group

Group decoding Literal items:

1. Store in hash table2. Copy

Copy items:1. Decode2. Replace3. Copy

Proposed SolutionHeader decoding Determine file size and if it is compressed or not.

Control bytes decoding Determine number and location of copy items.

Group decoding Literal items:

1. Store in hash table2. Copy

Copy items:1. Decode2. Replace3. Copy

EOFFile decompressed!

Filecompressed

File notcompressed

Next group

Decompression example

Decompression exampleHeader decodingThe header is composed of 8 bytes:

Meaningless zeroes

Decompression exampleHeader decodingThe header is composed of 8 bytes:

File size information: SIZE = (0 x 256) + 35 + 4 = 39

Byte 3 Byte 4

Decompression exampleHeader decodingThe header is composed of 8 bytes:

Compression indicator:0 = compressed1 = not compressed

Decompression example

Meaningless zeroes

Header decodingThe header is composed of 8 bytes:

Decompression exampleControl Bytes AnalysisThe control bytes are consisted of 2 bytes, each bit represent an item in the group:

Decompression exampleControl Bytes AnalysisThe control bytes are consisted of 2 bytes, each bit represent an item in the group:

0 – A literal item in the appropriate group position1 - A copy item in the appropriate group position

Decompression exampleControl Bytes AnalysisThe control bytes are consisted of 2 bytes, each bit represent an item in the group:

0 – A literal item in the appropriate group position1 - A copy item in the appropriate group position

Decompression exampleControl Bytes AnalysisThe control bytes are consisted of 2 bytes, each bit represent an item in the group:

0 – A literal item in the appropriate group position1 - A copy item in the appropriate group position

Decompression exampleData Bytes AnalysisBytes 11-22: literal item. Sent to hash function and copied as is to output file.

Decompression exampleData Bytes AnalysisBytes 11-22: literal item. Sent to hash function and copied as is to output file.

Decompression exampleData Bytes AnalysisBytes 11-22: literal item. Sent to hash function and copied as is to output file.

Compressed stream

Hashfunc.

Offset

Hash table

Decompression exampleHashing

The decompressor maintains a hash table identical to the one created by the compressor.

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

Length = 0 + 3

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

The number of copied literals is 3.

The offset is found in slot 3645 in the hash table.

Length = 0 + 3

Decompression exampleData Bytes AnalysisBytes 23-24: copy item

The number of copied literals is 3.

The offset is found in slot 3645 in the hash table.

Length = 0 + 3

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Serial compressed data

115200 bit/sec

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Parallel data (bytes)

wishbone protocol125MHz

REUSE

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORECompresseddata bytes 125 MHz

REUSE REUSE

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Decompresseddata bytes 125 MHz

REUSE REUSE

REUSE

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Decompressed bytes

wishbone protocol 125 MHz

REUSEREUSE

REUSE

REUSE

Project Top Block DiagramWM-

1

WS-3WM-

3

WS-2

WS-1

WM-2

LZRW3DECOMPRESSION

CORE

Serial decompressed data

115200 bit/sec

REUSE REUSE

REUSE

REUSE

Load uncompressed file PROCESS & COMPARE

Input size

x

Output (golden model):Messages (progress): (sending packets to FPGA/ decompressing/

getting files back/ comparing results)

(browse)

RAW INPUT

COMPRESSED INPUT

STATUS

Planned GUI

Load expected output file (browse)

DECOMPRESSLoad compressed file (browse)

COMPARE

Manually insert data PROCESS & COMPARE

Date Goals

21/3/2014 – 5/4/2014 Project Characterization &Algorithm interpreting

6/4/2014 Characterization Presentation

7/4/2014 – 20/4/2014 Full Characterization of all blocks

21/4/2014 – 10/6/2014 •System blocks VHDL •Design

11/6/2014 Mid presentation

11/6/2014 – 25/6/2014 Building a simulation environment

26/6/2014 – 25/7/2014 Work on project paused for exams

Project Schedule 1/2

Date Goals

30/7/2014 – 3/9/2014 Simulation run & debug

4/9/2014 Part A - Final presentation

21/9/2014 – 5/10/2014 FPGA synthesis & implementation

6/10/2014 – 23/10/2014

GUI implementation

24/10/2014 – 8/11/2014

Tests & debug

9/11/2014 – 23/11/2014

Writing project portfolio

24/11/2014 Final project presentation

Project Schedule 2/2

Weeks: 0 - 5 6 – 12 13 – 19 20 - 26 27 - 32

Characterization & interpretation

Characterization presentation

Blocks characterization VHDL blocks

implementation Mid presentation Simulation

environment Exams Simulation – Cont. Part A - Final pres. FPGA synthesis GUI implementation Tests & debug Writing portfolio Final presentation

.........……

……………...………..

............….……………………………………………………….…….…

44

7

2

2

2

2

3

Project Gantt

2

.……………………………………….…….…