ece 5578 multimedia communication lec 01 - introduction to ...2) media transport:...

ECE 5578 Multimedia Communication Lec 01 - Introduction to Image and Video Coding

Zhu LiDept of CSEE, UMKC

Office: FH560E, Email: lizhu@umkc.edu, Ph: x 2346.http://l.web.umkc.edu/lizhu

ECE 5578 Multimedia Communication, 2020. p.1

slides created with WPS Office Linux and EqualX equation editor

Outline

Background Objective of the class Prerequisite Lecture Plan Course Project Q&A

ECE 5578 Multimedia Communication, 2020. p.2

Video Growth

Better and Bigger Video: Moving to UHD 4K resolution HDR – High Dynamic Range Much higher data rate

Mobile Video: 25X growth predicted over the next 5 years

(source Cisco Visual Networking Index) Low power encoding/decoding on devices Mobile network capacity gap

Internet Video: Already accounts for more than half the

internet traffic Netflix alone 70% of ISP traffic at peak

time. Need better coding efficiency

ECE 5578 Multimedia Communication, 2020. p.3

Video Signal

ECE 5578 Multimedia Communication, 2020. p.4

Time

Video Compression

ECE 5578 Multimedia Communication, 2020. p.5

[Source: ISCAS 2014 tutorial, V.Sze and M. Budagavi]

Compression Approaches Basics of Video Compression Remove redundancy from the signal – brain to eye connection is

estimated to have only several hundred bits/sec capacity.

Video Signal Redundancy: Spatial Redundancy Perceptual Redundancy Temporal Redundancy Symbol Statistical Redundancy

ECE 5578 Multimedia Communication, 2020. p.6

Spatial Redundancy Removal

Intra Prediction – predict within the frame

ECE 5578 Multimedia Communication, 2020. p.7

Spatial Redundancy Removal

Block Transform:

ECE 5578 Multimedia Communication, 2020. p.8

Perceptual Redundancy

Human Visual System (HSV) is more sensitive to changes in low freq. areas than high freq. Capitalize on this by allow for coarser quantization at different

texture areas.

ECE 5578 Multimedia Communication, 2020. p.9

Perceptual Redundancy Removal

Quantization

ECE 5578 Multimedia Communication, 2020. p.10

Quant table

Quantized DCT coeff

Temporal Redundancy

Exploit redundancy among successive frames

ECE 5578 Multimedia Communication, 2020. p.11

Naïve !

Motion Compensation

Block based motion compensation for Temporal Redundancy removal

ECE 5578 Multimedia Communication, 2020. p.12

Video Motion Prediction Structure

I, P, B, b Frames…

ECE 5578 Multimedia Communication, 2020. p.13

Symbol Stats Redundancy

Not all pixel values have the same frequency Say if 128 appears 90% of time, we can use short bits to represent, and

longer bits to represent the rest of symbols Entropy coding !

ECE 5578 Multimedia Communication, 2020. p.14

Video Compression in Summary

ECE 5578 Multimedia Communication, 2020. p.15

Video Coding Standard

Why Standard ?

ECE 5578 Multimedia Communication, 2020. p.16

Video Coding Standard History

Pre-HEVC

ECE 5578 Multimedia Communication, 2020. p.17

Rate-Distortion Performance

Pre-HEVC

ECE 5578 Multimedia Communication, 2020. p.18

HEVC – High Efficiency Video Coding

Objective of HEVC

ECE 5578 Multimedia Communication, 2020. p.19

HEVC History

Very brutal…

ECE 5578 Multimedia Communication, 2020. p.20

HEVC Performance

ECE 5578 Multimedia Communication, 2020. p.21

HEVC Resources

Overview Paper: G. J. Sullivan, et al. "Overview of the High Efficiency

Video Coding (HEVC) standard,” IEEE Transactions on Circuits and Systems for Video Technology, 2012

Spec: H265 High Efficiency Video Coding Specification http://sist.sysu.edu.cn/~isscwli/ref/h265.pdf

Reference Software: – https://hevc.hhi.fraunhofer.de/svn/svn_HEVCSovware/

Python Tool: HARP http://lms.lnt.de/HARP/

ECE 5578 Multimedia Communication, 2020. p.22

New Media – 3D Image/Video

Point Cloud Capture & Compression

ECE 5578 Multimedia Communication, 2020. p.23

Passive Camera Array Active Lidar/Radar

Point Cloud Capture and Compression

Use Cases Content Capture for VR/AR

o Replay Technology (Acquired by Intel)o 8i.com

Navigationo Point cloud map for navigation

Challenges Point Cloud Static and Dynamic Geometry

Compression Point Cloud Attributes Compression

ECE 5578 Multimedia Communication, 2020. p.24

Video-based point cloud compression

Basic steps Normal-based projection, frame packing, and frame padding

Normal-based projection Organize the points with similar normal into a patch Project each patch to the 3D point cloud bounding box

p.25

Video-based point cloud compressionFrame packing: pack the patches into frames Exhaustive search empty space for the current patch Patch rotation is supported Introduced a lot of sharp edges

p.26

VPCC - Texture PaddingTexture padding: a number of methods are proposed to minimize

the bitrate of the unoccupied pixelsUsing push-pull algorithm as an example, like dilation

p.27

Video-based point cloud compressionBasic idea: project a point cloud to a 2-D video for an efficient

compression

p.28

Geometry

Attribute

Video Communication

The Dimension of the Video Communication Problem: Content: managed vs UGC Transport: managed IP networks vs Over The Top (OTT) networks Devices: managed settop boxes, vs unmanaged devices

ECE 5578 Multimedia Communication, 2020. p.29

[Christian Timmerer & Ali Begen, ICME 2015 Tutorial]

Current Landscape

Managed vs Unmanaged:

ECE 5578 Multimedia Communication, 2020. p.30

[Christian Timmerer & Ali Begen, ICME 2015 Tutorial]

Traditional TV vs OTT Video

Services are converging…

ECE 5578 Multimedia Communication, 2020. p.31

Netflix Phenomena

Purely OTT Solution, very successful in user penetration Accounted for a large portion of ISP traffic between 8-10pm.

ECE 5578 Multimedia Communication, 2020. p.32

HBO

ECE 5578 Multimedia Communication, 2020. p.33

Internet is dominated by video traffic

Top Down/ Bandaid approach: CDN (Content Delivery Network) Solutions, mostly based on HTTP

transport

New Trend/Clean Slate Approach: ICN (Info Centric Networking), design from layer 3 up to support

video

ECE 5578 Multimedia Communication, 2020. p.34

PSS over managed IP networks

Managed mobile core IP networks

ECE 5578 Multimedia Communication, 2020. p.35

MPEG DASH – OTT

HTTP Adaptive Streaming of Video

ECE 5578 Multimedia Communication, 2020. p.36

Quality of Experience (QoE)

ECE 5578 Multimedia Communication, 2020. p.37

QoE Spatial quality Temporal quality Viewing condition Packets received

Prerequisite & Text book

Prerequisite Good C and Python programming skills Taken Digital Signal Processing and/or Digital

Image Processing, or consent of the instructor Will have different expectation for MS and

undergrad students

Textbook: None required (saving $$) , will distribute relevant

chapters, papers, and notes.

Key References: Y. Wang, J. Ostermann, and Y.Q.Zhang, "Video Processing and

Communications," 1st ed., Prentice Hall, 2002. ISBN: 0130175471. K. Sayood, Introduction to Data Compression, 3rd Edition

ECE 5578 Multimedia Communication, 2020. p.38

Tentative Plan

Video Coding1) Entropy and Info Theory Background2) Lossless Coding/Entropy Coding: Hoffman, Arithmetic3) Lossy Image Coding: Transforms, Quantization, Wavelet/JPEG20004) Video Signal Processing: Sampling, Motion Compensation5) Scalability and Super Resolution6) Video Coding Standards: HEVC

Video Systems/Networking1) QoE metrics,Source-Channel Coding and Error Resilience2) Media Transport: RTP/RTSP/RVSP/RTCP, HTTP/Websocket, WebRTC3) Congestion Measurement and Control: TCP, SCTP, SPDY, NADA4) CDN, HTTP Cache, Cache Deduplication and delivery acceleration5) OTT Video Streaming Standards: DASH, MMT6) Overlay networks and P2P video streaming

ECE 5578 Multimedia Communication, 2020. p.39

Course Outcome

Upon completion of the course you will be able to: Understand the basic compression theory and algorithms for media

compression. Familiar with the latest media compression and communication

solutions and have hands on experiences. Can apply the knowledge an algorithms to solve real world media

communication problems Have good job and internship prospect with both Semi Conductor

Makers like QCOM, MediaTEK, Samsung, Intel, Broadcom, and content provider and CDN/ISP operators like AKAMAI, Netflix, Youtube, COMCAST, HBO, Sprint, ATT.

ECE 5578 Multimedia Communication, 2020. p.40

Grading

Homeworks (40%) Lossless coding of symbols Lossy image coding Motion Compensation Video codec excercise Network simulation of video streaming

Quiz (30%) : relax, quiz is actually on me, to see where you guys stand Quiz-1: On Video Coding Quiz-2: On Video Communication

Project (30%) Original work leads to publication (1.25x factor), discuss with me by

Spring break Regular project: assign papers to read, implement certain aspect, and

do a presentation

ECE 5578 Multimedia Communication, 2020. p.41

Logistics

Office Hour: Tue: 4:00-5:30pm, 560E FH Or by appointment

TA: TBD Office Hour/Lab Sessions are planned to cover certain software tools

aspects. Office Hour: TBA

Course Resources: Will share a box.com folder with slides, references, data set, and

software

ECE 5578 Multimedia Communication, 2020. p.42

Q&A

…..

ECE 5578 Multimedia Communication, 2020. p.43