content adaptive encoding method for high frame …yaqintang.github.io/files/thesisslides.pdfcontent...
TRANSCRIPT
![Page 1: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/1.jpg)
Content Adaptive Encoding Method
for High Frame Rate
Screen-Camera Communication
M.S. Defense Presentation
Yaqin Tang
Advisor(s): Prof. Marco Gruteser,
Prof. Kristin Dana, Prof. Narayan Mandayam
January 13th 2016
WINLAB, Rutgers University
![Page 2: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/2.jpg)
Screen-camera Communication
2
Transmitter (screen) Receiver (camera)
1 0 1 …
Image analysis
Signal demodulationSignal modulation
1 0 1 …
![Page 3: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/3.jpg)
Motivations and Objectives
3
User experienceInvisibility
Communication performanceGoodput (accurate bits per unit time)
Key considerations:
20
15
10
5
0noticeable unnoticeable
Flicker / distortion
Go
od
pu
t(k
bp
s)
Existing worksOur Goal
![Page 4: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/4.jpg)
Outline
4
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 5: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/5.jpg)
Outline
5
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 6: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/6.jpg)
Flicker Perception
6
Definition: apparent fluctuation and change in the
brightness of the displaying surface.
Affecting factors:
Frame rateModulation
amplitude
Image
content
Edge effect
Viewer’s
field of view
![Page 7: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/7.jpg)
Flicker Perception – frame rate
7
No obvious flicker if:
Brightness change frequency > 100 Hz.
light source
observer
lens
image light
control frequency
* Note: this is a simplified experiment setup from D.H. Kelly, “Sine wave and flicker fusion”.
D
O
U
![Page 8: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/8.jpg)
Flicker Perception – modulation amplitude
8
Signal amplitude experiment:
- x: original brightness;
- α, β: alteration amplitude.
x x + α x x - β
Frame 1 Frame 2 …
![Page 9: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/9.jpg)
Flicker Perception – modulation amplitude
9
Brute force method:• check brightness from (0~255) ± (1~10).
x ~ (0, 255)
α, β 1 2 3 4 … 9 10
1
2
3
…
10
• (+2 / -3) win for flicker perception and camera detection.
![Page 10: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/10.jpg)
Flicker Perception – edge effectviewer’s field of view
10
xx + α
x - β
Observation:
• Along the edges more obvious flicker.
• Smaller field of view in retina less flicker.
(combine viewing distance and display block size)
Gray block experiment
![Page 11: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/11.jpg)
Flicker Perception – image texture
11
• Attribute representing spatial arrangement of gray levels
of the pixels in a region of image.
• Texture regions give less flicker.
![Page 12: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/12.jpg)
Flicker Perception – brightness and contrast
12
Image brightness:• Visual perspective, color in R, G, B space or gray scale
from 0 to 255 pixel intensity value.
Image contrast:• Visual concept defined by the difference in the color and
brightness of the object.
Both are minor factors!
![Page 13: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/13.jpg)
Flicker Perception Factors
13
Frame rate -> greater than 100 Hz.
Modulation amplitude -> (+2 / -3) brightness alternation.
Image content -> image texture (major factor)
image brightness / contrast (minor)
Edge effect -> more flicker along edges.
Viewer’s field of view -> smaller area in retina, less flicker
![Page 14: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/14.jpg)
Outline
14
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 15: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/15.jpg)
Content Adaptive Encoding Method
15
Temporal domain encoding:
• Message bit-stream
assigned to each frame;
• Image modulated as
brightness change.
1 0 0 1 1 0 …
Displaying video
1
11
00
Consecutive same
bits reduce
displaying
frequency!
![Page 16: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/16.jpg)
Content Adaptive Encoding Method
16
Temporal domain encoding:• Manchester code ensures minimum frequency at 60 fps.
• Bit 1, brightness increase 2; bit 0, decrease 3.
![Page 17: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/17.jpg)
Content Adaptive Encoding Method
17
Spatial domain encoding:• Checkerboard on top of each frame.
- improve throughput;
- decease field of view in retina.
• Texture and edges analysis.
![Page 18: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/18.jpg)
Content Adaptive Encoding Method
18
Image texture analysis:• Texture range filter.
2 3
Min
4 5
7 8 9 10
12 13 14 15
Max
17 18 19 20
5 7 7 6
11 12 12 11
11 12 12 11
6 7 7 5
Texture range value:
15 – 3 = 12
Original image matrix sample
Corresponding
texture range value
![Page 19: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/19.jpg)
Content Adaptive Encoding Method
19
Image texture analysis:• Texture range filter.
• Choose a band pass filter to:
- get texture region & avoid edges
Band-pass filter
![Page 20: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/20.jpg)
Content Adaptive Encoding Method
20
![Page 21: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/21.jpg)
21
Our encoding method Reasons
Temporal encoding
• Display at 120fps;
• Brightness change +2, -3 with
Manchester code;
Frame rate
Modulation amplitude
Spatial encoding
• Checkerboard size 32*32 pixel²;
• Texture range analysis and
bandpass filter.
Viewer’s field of view
Edge effect
Image texture
Content Adaptive Encoding Method
Flicker and goodput oriented:
![Page 22: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/22.jpg)
Outline
22
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 23: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/23.jpg)
Decoding Method
23
Assumption, receiver knows:• Starting frame of the message;
• Original video resolution;
• Checkerboard size.
• Encoded checkerboard patterns;
Algorithm:• Track temporal brightness change, if
- High to low “bit 1”
- Low to high “bit 0”
![Page 24: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/24.jpg)
Decoding Method
24
![Page 25: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/25.jpg)
Frame Alignment Issue
25
Clean
frames!
![Page 26: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/26.jpg)
Frame Alignment Issue
26
Histogram method to extract clean frames.
Larger peak distance
![Page 27: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/27.jpg)
Encode Pattern Detection
27
Recap: - content-adaptive encoding
- selected blocks encoded message.
Algorithm: find large brightness change blocks.
![Page 28: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/28.jpg)
Outline
28
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 29: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/29.jpg)
Prototype Implementation
29
• 120fps video displayed using glvideoplayer.
• Iphone6 recording at 240 fps.
Video resolution
1280-by-720
Camera fixed
auto-exposure;
in Slo-Mo mode
![Page 30: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/30.jpg)
Experiment Videos
30
![Page 31: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/31.jpg)
Outline
31
• User experience: flicker perception factors
• System design
- content-adaptive encoding method;
- signal amplitude tracking decoding method;
• Prototype implementation
• System performance evaluation
• Conclusion and future work
![Page 32: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/32.jpg)
Pattern Detection Evaluation
32
• Static scene, color videos
![Page 33: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/33.jpg)
Sample Error Markers
33
Red
not encoded
blocks as
encoded;
Green
bit 0 blocks as
not encoded;
Blue
bit 1 blocks as
not encoded;
Gray
original
encoded
blocks without
error.
![Page 34: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/34.jpg)
Basic Decoding Algorithm Evaluation
34
• Static scene, color videos Goodput: correct bits per unit time.
![Page 35: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/35.jpg)
Basic Decoding Algorithm Evaluation
35
• Dynamic scene, color videos Goodput: correct bits per unit time.
![Page 36: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/36.jpg)
Conclusions and Future Works
36
• Explored factors contributing to flicker perception;
• Proposed content-adaptive encoding method to achieve
flicker-free screen-camera communication as well as high
communication capacity and accuracy;
• Identified reasons causing system error;
• Combine pattern detection to decoding algorithm;
• Applications like screen identification etc.
![Page 37: Content Adaptive Encoding Method for High Frame …yaqintang.github.io/files/thesisslides.pdfContent Adaptive Encoding Method for High Frame Rate Screen-Camera Communication M.S. Defense](https://reader034.vdocument.in/reader034/viewer/2022042808/5f862e1fa7fdc707b81a9756/html5/thumbnails/37.jpg)
Thank you!
37