The Visual Microphone: Passive Recovery of Sound from Video

Copyright of figures and other materials in the paper belongs original authors.

Presented by Ki-hoon Kim

2015.09.09

Computer Graphics @ Korea University

Abe Davis et al.SIGGRAPH 2014

Ki-hoon Kim | 2016-01-07 | # 2Computer Graphics @ Korea University

3D Position Measurement From Image

Image Data

3D Geometry Data

Ki-hoon Kim | 2016-01-07 | # 3Computer Graphics @ Korea University

Emotion Recognition

Image Data

Sound DataEmotion

Recognition

Ki-hoon Kim | 2016-01-07 | # 4Computer Graphics @ Korea University

Introduction

Ki-hoon Kim | 2016-01-07 | # 5Computer Graphics @ Korea University

• Recording the reflection of a laser pointed at the surface.

Record phase of a reflected laser

Recover high quality audio

Depend on precise positioning(Laser & Receiver)

Surface must be appropriate reflectance

Related Work - Laser microphone

Ki-hoon Kim | 2016-01-07 | # 6Computer Graphics @ Korea University

Using high-speed camera

Record changes in the speckle pattern of laser light

Greater flexibility in the positioning of a receiver

Depend on recording reflected laser light

Related Work - Laser microphone

Heart beat Human Voice

Simultaneous remote extraction of multiple speech sources and heart beats from secondary speckles pattern[Zalevsky et al./Opt. Express 2009.]

Ki-hoon Kim | 2016-01-07 | # 7Computer Graphics @ Korea University

Related Work – Extract subtle motions

Ki-hoon Kim | 2016-01-07 | # 8Computer Graphics @ Korea University

Overview

Ki-hoon Kim | 2016-01-07 | # 9Computer Graphics @ Korea University

• 0. Decompose the input video into spatial sub-bands

Corresponding to different orientations and scales

• 1. Computing the Local Motion Signals(Phase Variations)

• 2. Computing the Global Motion Signal

• 3. Denoising

Recovering Sound from Video

Ki-hoon Kim | 2016-01-07 | # 10Computer Graphics @ Korea University

• Complex Image can be expressed(Riesz Pyramid Transform)

𝐴 : Amplitude

𝜑 : Phase

𝜃 : Transform Orientation

𝑟 : Transform Scale

• Compute phase variations

Recovering Sound from VideoLocal Motion Signals

Ki-hoon Kim | 2016-01-07 | # 11Computer Graphics @ Korea University

• Weigh each local signal by its squared amplitude

• Shift in time to align phase signals

• Global motion signal is

Recovering Sound from VideoGlobal Motion Signal

Ki-hoon Kim | 2016-01-07 | # 12Computer Graphics @ Korea University

• Modeling object motion

(a,b) 300 Hz pure tone, [0.1,1] Pa

(c) 20 Hz to 2,200 Hz frequency

Object Response

Ki-hoon Kim | 2016-01-07 | # 13Computer Graphics @ Korea University

• Modeling pixel motion

𝑚 : Magnification of surface in [mm/pixel]

𝜃 : Viewing angle of camera

𝜔 : Frequency

• SNR of recovered sound relationship

𝑛𝑝 : The number of pixels

𝜎𝑛 : Image Noise Standard Deviation

Processing

Ki-hoon Kim | 2016-01-07 | # 14Computer Graphics @ Korea University

Experiments - Frequency Range

Input Sound

ReconstructedSound

Ki-hoon Kim | 2016-01-07 | # 15Computer Graphics @ Korea University

Experiments – Recovery Comparison

• VM – Our Method, LDV – Laser Doppler Vibrometer

• Bold: Better score

• VM is comparable to LDV

• LDV is required the retro-reflective tape

Ki-hoon Kim | 2016-01-07 | # 16Computer Graphics @ Korea University

Recovering Sound with Normal Video Cameras- What is Rolling Shutter?

Ki-hoon Kim | 2016-01-07 | # 17Computer Graphics @ Korea University

Recovering Sound with Normal Video Cameras- How to use Rolling Shutter

• aaaa

Ki-hoon Kim | 2016-01-07 | # 18Computer Graphics @ Korea University

• Assume exposure time 𝐸 ≈ 0,

𝐼𝑛 : Rest State Image

𝐵𝑛 : Image with Rolling Shutter Effect

𝑛 : Frame number

𝑇 : Frame Period

𝑑 : Line Delay

Recovering Sound with Normal Video Cameras- Equation Modeling

Ki-hoon Kim | 2016-01-07 | # 19Computer Graphics @ Korea University

Recovering Sound with Normal Video Cameras- Convert to audio signal

• Missing Sample by Frame Delay

Audio interpolation

Ki-hoon Kim | 2016-01-07 | # 20Computer Graphics @ Korea University

Results

Ki-hoon Kim | 2016-01-07 | # 21Computer Graphics @ Korea University

Q&A