1

H.264 Analyzer Utility Tool

ByMukta GoreShantanu KurhekarAmruta VaidyaGouri Patil

2

Project Statement

n Decoding the H.264 input bit stream step wise, restricting its purview to the Baseline profile.

n Extracting NAL units from the bit stream, and displaying list of frames in the video sequence along with list of parameter sets encoded in the stream.

n Decoding and displaying the syntax elements in the parameter sets.

n Displaying user selected frame.

n Extracting selected slice in selected frame, and decoding and displaying parameters in the slice header.

n Decoding and displaying selected macroblock information.

3

Analyzersn A software tool to help technical professionals parse bit streams

conforming to international media standardsn Video standards specify only a syntactic description of compressed

visual data and method for decoding it to reconstruct video sequence

n Since encoding methods are not specified, analyzers are needed to test for conformance to the standards

n Analyzers perform stepwise decoding of compressed bit streams and display results at every step to check for non-conforming parameters

n They also help in testing the working and efficiency of algorithms for motion estimation, rate control and scene change detection

4

A peek into our utility …

5

H.264 ENCODER

6

H.264 DECODER

7

Implementation of our Analyzer

Library

API

User Interface

User Interface : GUI in Visual Basic

API : Application Program Interface

Library : Dynamic Link Library in C

8

User Interfacen This acts as an interface between the user and the core functionality

of the analyzer n It enables user-controlled picture-by-picture analysis of .264 streamn The user can analyze the bit stream at various levels in the

hierarchy-sequence, frames, slices, macroblocks and macroblockpartitions

n It displays the parameter sets, reconstructed sample values, errors in the bit stream, if any and the reconstructed picture

9

Parameter listSlice list

Frame list

Macroblock list

MB info choice

Detailed MB info based on choice

selected

Reconstructed frame

Option to Hide grid

10

Libraryn We have implemented the core decoding functions as a Dynamic

Link Library because

¨ The analyzer uses decoding functions written in C which is roughly a 14000 line code

¨ DLLs link the object code to the executable at run time rather than compile time as done in static libraries

¨ So less memory is required to run the analyzer utility, and the executable file is compact

¨ Multiple applications can share the same DLL¨ Multiple entry points are provided in DLLs to enable conditional

decoding

11

Program Flown The functionality of the utility tool is event driven so depending on the

action of the user, various tasks are performed.n Processing is done at two levels: GUI level and DLL leveln Following events actuate entry into the DLL –

¨ Opening a file by clicking on the submenu Open in the menu File¨ Clicking on a frame in the frame list¨ Clicking on a slice in the slice list¨ Closing a file by clicking on the submenu Close in the menu File

n Certain events like ¨ clicking on SPS or PPS in the parameter list¨ clicking on a macroblock in the MB list ¨ selecting MB info choiceare handled in the GUI domain without calling the DLL again

12

Open File

PRIME Prime

Extract NALunit

Parse

GUI DLL

Display No. of

NAL units,

parameter set list

and frame list

Event 1: Clicking on submenu Open of menu File

13

Extract NAL unit Parse

14

NAL unit structure

Start code NAL Header Raw Byte Sequence Payload

NAL unit

0x000001

Bit 0 Forbidden bit Value:0

Bits 1-2 NAL ref idcValue:0-3

Bits 3-7 NAL type Value:0-31

Video Coding Layer (VCL) data

15

Event 2: Clicking on an SPS from list Event 3: Clicking on a PPS from list

Parameter set mechanism enables an encoder to signal important, infrequently changing sequence and picture parameters separately from the coded slices

Sequence Parameter Set is applicable to a complete video sequence and contains profiletype, the number of reference frames, decoded picture width and height, choice of progressive or interlaced coding etc

Picture Parameter Set is applicable to one or more pictures within a sequence and contains selected SPS id, flag to select VLC or CABAC entropy coding, number of slice groups in use, number of reference pictures in list 0 used for prediction, initial quantization parameters etc

16

Event 4: Clicking on a frame from the list

17

Frame decoding

n The following process will be iterated till all slices in the selected frame are decoded

n I or IDR frame:¨ It contains only intra coded macroblocks and hence is decoded

independently of previous pictures¨ The starting point is first slice of the frame

n P frame:¨ It contains inter and intra coded macroblocks and hence its reference

pictures need to be decoded¨ The starting point is first slice of the most recent IDR¨ To improve efficiency,

if (recent IDR < last decoded frame < frame to be decoded)then starting point is first slice of last decoded frame

elsestarting point is first slice of recent IDR

18

Slice decoding

n Slice header¨ Slice type¨ Picture Parameter Set referenced¨ Frame number to which it belongs with respect to recent IDR¨ Delta quantization parameter at slice level¨ No. of active reference pictures in list 0

n Slice data¨ Macroblock layer

Slice Header Slice data

19

Macroblock layer

n MB type (Inter/Intra/Skipped)n Macroblock partitions for prediction

¨ Intra – 4x4, 16x16, PCMn 9 prediction modes for 4x4n 4 prediction modes for 16x16

¨ Inter – allows sub-block partitioning up to 4x4n Separate motion vector for each partition or sub-partitionn Quarter pel motion vector accuracyn Separate reference picture for each macroblock partition allowed

n Delta quantization parameter at macroblock leveln Coded block pattern to indicate which of the blocks contain non-zero

coefficients

20

Event 5: Clicking on a slice from the list

21

Event 6: Clicking on an MB from the list

The information extracted during frame decode is accessed in GUI domain for display, without calling DLL

22

Event 7: Clicking on submenu Close of menu File

23

Error handlingn The utility checks if all parameters are within their specified range to

test conformance with H.264 baseline profilen Bit stream errors are handled at 4 levels

¨ General - Unsupported file format, No SPS, No PPS, No IDR¨ SPS,PPS¨ Slice header¨ Slice data

n On return from DLL, the error array is checked and correspondingerrors are displayed with the relevant frame number

24

Challengesn Datatype matching of structures in DLL and GUI domainn DLL convention for parameter passing n Accessing memory allocated in DLL from GUIn Finding out reference frame for every macroblock partitionn Memory allocation and deallocation problems

25

Utility Featuresn User-controlled picture by picture analysis n Supports CIF (352 x 288), QCIF (176 x 144), SQCIF (128 x 96) video formatsn Stream level statistics collectionn Only Baseline profile supportedn Conformance checks (Display of non conformant parameters)n Parameter set extraction facility (PPS, SPS) n Frame Level Analysis

¨ Frame type¨ Number of slices in the frame¨ Distribution of Macroblocks in the frame based on type

n Slice Level Analysis¨ Slice Header¨ Number and address of Macroblocks in the slice

n Macroblock Level Analysis¨ Macroblock type, subtype and prediction mode¨ Quantization parameters for luma and chroma¨ MB grid overlay on picture with Hide Grid option¨ Motion vector display - MB and sub-MB partitioning

26

Applications of analyzern Getting insight into the coding quality of H.264 / MPEG-4 part 10

video-encoding software and hardware n Enabling rectification of the functions responsible for encoding non-

conformant parameters in the encodern Analysis of coded bit stream for no. of I and P frames, macroblock

and sub-block partition for rate controln Monitoring and validating large volumes of digital media down to

inspecting and debugging any problems encountered at the bitstream level

27

Files tested

High motion activity video of 90 frames, 1 slice per frame, CIF sizeStefan.2641 : 27

High motion activity video of 90 frames (1 I and 89 P), 1 slice per frame, CIF size

Football.2641 : 31

Medium motion activity video of 90 frames (1 I and 89 P), 1 slice per frame, QCIF size

Carphone.2641 : 90

Low motion activity video of 97 frames (1 I and 96 P), 1 slice per frame, CIF size

News.2641 : 158

Low motion activity video of 60 frames (1 I and 59 P), 3 slices per frame, QCIF size

Md_33.2641 : 208

Medium motion activity video of 150 frames (1 I and 149 P), 1 slice per frame, QCIF size

Silent.2641 : 118

Low motion activity video of 150 frames (10 I and 140 P), 1 slice per frame, QCIF size

Akiyo.2641 : 168

Brief AnalysisFile name&

Compression

28

No IDR, no SPS, no PPS in bit streamtest.264Illegal luma ac coefficients (Frame 3)container_wipe.264

Weighted prediction and weighted bi-prediction flag not zero

md.264Profile not supportedforeman.264Unsupported file formatnoStartCode.264

ERROR DISPLAYEDFILENAME

Non - conformant bit streams

29

Future Scopen Adding support for Main and Extended profilen Displaying reference and current picture simultaneously for selected

macroblockn Visual display for motion vectors using arrowsn Slice group map typen Displaying the macroblocks in the order they are decoded if arbitrary

slice order is allowedn Support for SIF filen Support for simultaneous audio and video analysis

30