Download - Event-Based Computer Vision at Sony AVS
![Page 1: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/1.jpg)
Event-Based Computer Vision at Sony AVS
Christian Brändli, CEO, Sony AVS
CVPR 2021
![Page 2: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/2.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-212 © 2021 Sony Semiconductor Solutions Corporation
![Page 3: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/3.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-213 © 2021 Sony Semiconductor Solutions Corporation
So Who Is Sony Semiconductor?
Sony Group Corporation
Sony Interactive Entertainment
Sony Music Group(Global)(Sony Music Entertainment, Sony Music Publishing)
Sony Music Entertainment Japan
Sony Pictures Entertainment
Sony Corporation
Sony Financial Holdings
Sony Semiconductor Solutions
Game & NetworkServices
Music
Electronics Products &Solutions
Pictures
Imaging & SensingSolutions
Financial Services
![Page 4: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/4.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-214 © 2021 Sony Semiconductor Solutions Corporation
We Build The World’s Best Image Sensors
Image Sensor Sales
![Page 5: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/5.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-215 © 2021 Sony Semiconductor Solutions Corporation
Who is Sony AVS?
Design/development bases
Manufacturing bases
Sales bases
Sony Precision Devices (Huizhou)
(Huizhou)
Sony Device Technology (Thailand)
(Bangkadi)
Sony Depthsensing Solutions
(Brussel)
Sony Semiconductor Israel
(Israel)
San Jose
Seoul
Taipei
Hong Kong
Singapore
Weybridge
Shanghai
Image sensor assembly etc.Optical pickup
Sony Advanced Visual Sensing
(Schlieren)
Midokura Ibérica
(Barcelona)
Altair Semiconductor France
(France)
ALT Semiconductor Finland
(Finland)
Sony Semiconductor IsraelTaiwan Branch
(Hsinchu)
![Page 6: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/6.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-216 © 2021 Sony Semiconductor Solutions Corporation
And What Does Sony AVS Do?
We built and build Event-Based Computer Vision Algorithms
![Page 7: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/7.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-217 © 2021 Sony Semiconductor Solutions Corporation
Enough PR ...
... Let’s Talk Technology
![Page 8: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/8.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-218 © 2021 Sony Semiconductor Solutions Corporation
What Is A Temporal Contrast Event?
𝑒 = 𝑥, 𝑦, ෬𝑝, Ǎ𝑡 : ∃𝑒𝑥,𝑦[𝑗] ෬𝑝 =
1 𝑓𝑜𝑟 න𝑇 𝑗−1
ም𝑡
𝐶𝑥,𝑦𝑡 𝑡 + ሶ𝑁𝑥,𝑦 𝑡 𝑑𝑡 ≥ 𝛩𝑂𝑁
0 𝑓𝑜𝑟 න𝑇 𝑗−1
ም𝑡
𝐶𝑥,𝑦𝑡 𝑡 + ሶ𝑁𝑥,𝑦 𝑡 𝑑𝑡 ≤ 𝛩𝑂𝐹𝐹
A temporal contrast event is a tuple of an address with polarity ෬𝑝 and timestamp Ǎ𝑡.Such an event is created when …
… the integral since the last timestamp…
… of the temporal contrast…
… and the noise …
… is below …
…or above a threshold
![Page 9: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/9.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-219 © 2021 Sony Semiconductor Solutions Corporation
What is Temporal Contrast?
𝐶𝑥,𝑦𝑡 𝑡 =
1
൯𝐼𝑥,𝑦(𝑡
𝑑 𝐼𝑥,𝑦 𝑡
𝑑𝑡=
ቁ𝑑 (𝑙𝑛 𝐼𝑥,𝑦 𝑡
𝑑𝑡
Temporal contrast is the rate of illumination change…
… normalized by the absolute intensity… … which is equivalent to the log intensity change rate.
![Page 10: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/10.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2110 © 2021 Sony Semiconductor Solutions Corporation
So What is Encoded By an Event?
න𝑡0
𝑡1
𝐶𝑥,𝑦𝑡 𝑡 𝑑𝑡 = ∆𝐶𝑥,𝑦
𝑡 𝑡0, 𝑡1 = 𝑙𝑛 𝐼𝑥,𝑦 𝑡1 − 𝑙𝑛 𝐼𝑥,𝑦 𝑡0 = 𝑙 𝑛𝐼𝑥,𝑦 𝑡1𝐼𝑥,𝑦 𝑡0
Integrating temporal contrast results in …
… a “temporal contrast step”, …
… which is a fixed step in log intensity …
… or a fixed step factor.
![Page 11: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/11.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2111 © 2021 Sony Semiconductor Solutions Corporation
Then What Makes Up a Temporal Contrast Step?
𝐸𝑀𝑥,𝑦 𝑗 = ∆𝐶𝑥,𝑦𝑡 𝑡0, 𝑡1 = න
𝑇[𝑗−1]
𝑇 𝑗 𝑑𝑙𝑜𝑔(𝐼(𝑘, 𝑡))
𝑑𝑡+𝑑𝑅
𝑑𝑡−𝑑𝑅
𝑑𝑥
𝑑𝑥
𝑑𝑡−𝑑𝑅
𝑑𝑦
𝑑𝑦
𝑑𝑡𝑑𝑡
illumination change optical flowspatial contrast
reflectance change
The “event magnitude” is …
… the temporal contrast step encoded by one event, …
… which in turn is made up of different components.
![Page 12: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/12.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2112 © 2021 Sony Semiconductor Solutions Corporation
So What To Do With Events
𝐸𝑀𝑥,𝑦[𝑗] = න𝑇[𝑗−1]
𝑇 𝑗 𝑑𝑙𝑜𝑔(𝐼(𝑘, 𝑡))
𝑑𝑡+𝑑𝑅
𝑑𝑡−𝑑𝑅
𝑑𝑥
𝑑𝑥
𝑑𝑡−𝑑𝑅
𝑑𝑦
𝑑𝑦
𝑑𝑡𝑑𝑡
illumination change
reflectance change
optical flow
Entropy:- Smart Triggering- Smart Processing
Active Lighting:- Structured Light- Active Marker Tracking- Visible Light
Communication
?
spatial contrastevent magnitude
Reconstruction:- Imaging- MappingClassification:- Object
Tracking:- Camera:
- VIO- World:
- User- Objects
![Page 13: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/13.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2113 © 2021 Sony Semiconductor Solutions Corporation
Some Examples
![Page 14: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/14.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2114 © 2021 Sony Semiconductor Solutions Corporation
So What To Do With Events
𝐸𝑀𝑥,𝑦[𝑗] = න𝑇[𝑗−1]
𝑇 𝑗 𝑑𝑙𝑜𝑔(𝐼(𝑘, 𝑡))
𝑑𝑡+𝑑𝑅
𝑑𝑡−𝑑𝑅
𝑑𝑥
𝑑𝑥
𝑑𝑡−𝑑𝑅
𝑑𝑦
𝑑𝑦
𝑑𝑡𝑑𝑡
illumination change
Active Lighting
![Page 15: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/15.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2115 © 2021 Sony Semiconductor Solutions Corporation
High-Speed 3D
![Page 16: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/16.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2116 © 2021 Sony Semiconductor Solutions Corporation
So What To Do With Events
𝐸𝑀𝑥,𝑦[𝑗] = න𝑇[𝑗−1]
𝑇 𝑗 𝑑𝑙𝑜𝑔(𝐼(𝑘, 𝑡))
𝑑𝑡+𝑑𝑅
𝑑𝑡−𝑑𝑅
𝑑𝑥
𝑑𝑥
𝑑𝑡−𝑑𝑅
𝑑𝑦
𝑑𝑦
𝑑𝑡𝑑𝑡
optical flow
Tracking
![Page 17: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/17.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2117 © 2021 Sony Semiconductor Solutions Corporation
Very Fast Real-Time Tracking Of Dots
![Page 18: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/18.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2118 © 2021 Sony Semiconductor Solutions Corporation
Better For Fast Motion than Open CV
![Page 19: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/19.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2119 © 2021 Sony Semiconductor Solutions Corporation
And More Robust On Repetitive Structures
![Page 20: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/20.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2120 © 2021 Sony Semiconductor Solutions Corporation
So What To Do With Events
𝐸𝑀𝑥,𝑦[𝑗] = න𝑇[𝑗−1]
𝑇 𝑗 𝑑𝑙𝑜𝑔(𝐼(𝑘, 𝑡))
𝑑𝑡+𝑑𝑅
𝑑𝑡−𝑑𝑅
𝑑𝑥
𝑑𝑥
𝑑𝑡−𝑑𝑅
𝑑𝑦
𝑑𝑦
𝑑𝑡𝑑𝑡
spatial contrast
Reconstruction
![Page 21: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/21.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2121 © 2021 Sony Semiconductor Solutions Corporation
Sensor Fusion of iToF and EVS for Efficient Depth Sensing
𝑡0 𝑡1 𝑡2 𝑡3
Predict depth using events
𝑡0 𝑡1 𝑡2 𝑡3
Trigger depth less often → save powerHigh-speed depth
Replace depth measurements
Generate additional depth estimates
Predicted depth frames Predicted depth frames𝑡0 𝑡1 𝑡2 𝑡3
![Page 22: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/22.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2122 © 2021 Sony Semiconductor Solutions Corporation
To Capture What Our Engineers Are Doing In the Office
![Page 23: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/23.jpg)
Research Division 1 – Sony Advanced Visual Sensing AG16-Jun-2123 © 2021 Sony Semiconductor Solutions Corporation
If You Like To Work With The Latest And Greatest Sensors
https://linkedin.com/company/sony-advanced-visual-sensing
Follow us on LinkedIn:
Or Have A Look At Sony Jobs Every Now And Then:
https://www.sonyjobs.com/jobs.html
![Page 24: Event-Based Computer Vision at Sony AVS](https://reader033.vdocument.in/reader033/viewer/2022042221/625a30bac4bc9c05462eab5a/html5/thumbnails/24.jpg)