Optical Sensing: 1D to 3D using Time-of-Flight Technology

Shaping the Future of MEMS & SensorsSeptember 10, 2013

Optical Sensors Intro

Time-of-Flight Technology

ToF for 1D ranging

ToF for 3D gestures

Next Steps

Agenda

Marc Drader

Presentation

2

STMicroelectronics Imaging Division 3

Camera

Modules

Fixed focus cameraWafer Level re-flowable cameraEDoF cameraAuto-focus cameraInnovative optics, assembly & test technologies

Image

Sensors

Production from 1.4um to 5.6um pixel1.1um developmentFrom VGA to 24Mpix

Imaging

Processors

Stand alone ISPFull ST video pipe IPIntegration of third party IP on demand

Photonic

Sensors

User detection, Proximity, ALS, Optical navigation, Man Machine Interface, Automotive, Medical

Brief Overview: Optical Sensors 4

• May seem obvious but…

• Optical path considerations

• Transmission spectrum

• Transmission path efficiency

• Field of view

• Target object characteristics

• System considerations

• Optical crosstalk!

• Ambient or background illumination (noise)

Proximity Detection 5

• Conventional IR sensor �

• Attempt to detect whether object/user is near or far, based on reflected signal amplitude

• � impossible to know object DISTANCE

• Amplitude• Signal & Noise

1 Output

Noise

Far

Near

Threshold setting will detect user anywhere from 0.5 to 6cm

• Target Distance• Target Reflectance

2 Unknowns

90%

17%

3%

What is “Time-of-Flight” Sensing? 6

Active Illumination system:

1. Emit light (photons) towards a target

2. Light (partially) reflects from the target

3. Sensor determines “when” light (photons) arrive

Photon travel time multiplied by speed of light = distance

• 1cm = 66ps round-trip travel time at the speed of light

EmitterSensor

Target object

Photon(s)

Single photon travel time

Distance

Photon travel time NOT affected by target reflectance

ToF Illustration (video) 7

• Available on YouTube

Motivation for ToF

• Real-world application/need

• Best example: Smartphone proximity sensor detects user’s head during a phone

call; shuts off touchscreen & display

• But… it doesn’t work 100% of the time

• Search: “face hang-up” + any smartphone brand, to find frustrated users whose touchscreen did not shut off before their cheek pressed a button

• Time-of-Flight technology adds value by providing true, accurate distance measurements

• Independent of target object reflectance

• Immune to ambient illumination & optical path variations

(glass, plastic cover)

8

9Motivation for ToF

Dark hair

SkinS

ma

ll d

isp

lace

me

nts

Threshold

Threshold

Far(screen on)

Near(screen off)

Far(screen on)

Near(screen off)

• Conventional IR sensor bouncing between far and near

states vs robust ToF solution

• Photon travel time NOT affected by the object reflectance

Reflected power

(Conventional PS (*) )

ST ToF distance measurement

(*) Reflected power information also available on ST ToF Proximity Module

Time-of-Flight Physics

• Emit and receive photons :

• ….follow Poisson distribution

• …may be correlated or uncorrelated (ambient, dark current) to emitter

• Finally � photon arrival rate does depend on object reflectance & distance

10

= returned photons (correlated)

= ambient photons (uncorrelated)Emitted pulse

Delay we want to measure

Received pulse

(delay=distance)

���� Many repeated pulses required for correlation

Single Photon Avalanche Diode

• SPAD digital output used to:

• Count arrival of single photons and/or

• Time arrival of single photons

• Unique Properties

� each photon provides valuable time/distance info

• Fully Integrated in CMOS

11

More Time-of-Flight Challenges

• Optical constraints

• Coverglass contributes optical crosstalk (shortcut from emitter to sensor)

• Ambient light is main contributor of uncorrelated photons

• Co-existence of visible & NIR systems for ALS & ranging

• Conflicting wavelength and field-of-view requirements

12

Ambient light

Sensor field of view

IR emission

Phone Window

Airgap

Optical Crosstalk 13

• Time-of-Flight Sensor always “sees” two targets:

• Product-level cover (glass/plastic)

• fixed distance, and (relatively) fixed optical characteristics

• distorts reflected signal in both time & amplitude domain

• Target object

• varying distance and optical characteristics

EmitterSensor

Target object

Photon

�FlightSense technology compensates for optical crosstalk automatically

�Opens up use cases in very challenging optical environments

Crosstalk Compensation

• Compensation algorithm

• Firmware uses known crosstalk characteristics to correct the time-domain measure

• Simple register write (absolute value of photons from emitter to sensor coupled

through phone housing)

14

Raw range results (no compensation applied)

Raw range results (no compensation applied)

Crosstalk compensation applied (register setting)Crosstalk compensation applied (register setting)

Ambient Immunity

• System performance

• Keep ambient photons out

• Optical filtering (notch around 850nm)

• Reject remaining ambient photons

• Time-domain rejection

• System-level noise management

• SNR limit

15

�FlightSense technology will NOT report false distance in high ambient light conditions

• Ranging specifications

• 0 to 100mm, 3% to 90% reflectance

• 0 to 250mm for a 45% reflectance target (i.e.. Human hand)

• Eye-safe, low power IR (850nm) emitter

• Accuracy: σ = 3mm (resolution = 1mm steps)

• FlightSense architecture allows “zero” mm measurement

• * 0mm defined at the product/system-level

• There must be an available emitter�sensor optical path!

Ranging Conditions 16

Simple Optical Module 17

• Simple (reflowable) package

• Small size (2.8 x 4.8 x 1.0mm)

• Integrated emitter/sensor & optics/filters

• Opposing requirements

• Proximity: near infra-red wavelengths, narrow field of view

• Ambient Light Sensor: visible wavelengths, wide field of view

• Device delivered full calibrated

• Simple electrical integration

• Single power supply (2.8V)

• I2C & GPIO (1.8V or 2.8V)

• Programmable I²C address

• Flexible window & threshold interrupts

VL6180X Ranging Performance 18

Reflective charts (in %):

10x measurements per chart, 10mm step, in the dark,

0.2mm air-gap, no gasket, Oval artwork (75%>800nm)

• Ranging performance is independentof target reflectance/color

• Distance standard deviation < 3mm

Ranging performance is independent

of target reflectance/color

0

1000

2000

3000

4000

5000

6000

7000

8000

9000

10000

0 10 20 30 40 50 60 70 80 90 100

Co

nv

erg

en

ce T

ime

(u

s)

Target Object Distance (mm)

Convergence Time

3% 5% 17% 88%

VL6180X Convergence Time 19

Reflective charts (in %):

Distance (mm) vs Convergence Time (us)Target reflectance from 3% to 88%

0.00

0.50

1.00

1.50

2.00

0 20 40 60 80 100

Av

era

ge

Cu

rre

nt

(mA

)

Target Object Distance (mm)

Average Current Consumption10Hz repetition rate

88%

17%

5%

3%

Low Power Consumption

• Real-world current consumption

• Varies with object distance & reflectance

• Max consumption set by user

• Examples (2.8V supply)

• 10Hz ranging, object held @ 5cm

• 88% (white): 40µA

• 18% (grey): 200µA

• 5% (black): 550uA

• 3% (deep black): 760uA

• 1Hz ALS, 100ms integration

• ALS: 32uA average

• Low standby current

• HW standby <1uA

• SW standby <7uA

20

Conv. time

Conv. time

Peak current consumption

New FlightSense™ VL6180X Sensor 21

Fast, Accurate Distance Ranging• Independent of object reflectance (color)• Ambient rejection (sunlight, etc)• Phone window “crosstalk” compensation• Enable creative use cases (1D gesture application)

Disruptive Time-of-Flight Technology• 6 years of R&D• Key patents for innovative sensor/system architecture• Differentiating, unique technology• Manufactured in ST’s custom process

High-sensitivity ALS• “Invisible” for Industrial Design• Ultra-wide dynamic range• Calibrated output value in Lux

Simplified Integration & Manufacturing• Small reflowable module with embedded light emitter• No additional optics or gasket• No phone-to-phone calibration required• Robust to phone glass manufacturing dispersion • Robust to phone drop / minimize field return• Robust supply chain with dual sourcing strategy

Production in H1 2014

Product Readiness

• Mass Production in H1/2014

• CP code : VL6180XV0NR/1

• Not just for mobile phone applications

• Robust proximity detection

• Consumer robotics

• Gaming

• And much more!

• SPAD/ToF potential applications are endless

22

Check out our page on ST.com

ToF for Gestures: Motivation

• Multiple outputs eliminate ambiguity for gesture detection

23

Up/Down

Swipe

Distance

Distance

Amplitude

Amplitude

Multiple ToF sensor

Reduce ambiguity with more info:

• Order

• Position

24

1 device gestures capabilities : 2 devices gestures capabilities :

3D Gesture Detection

• High potential for differentiation

• Setting Expectations• Unlike a touchscreen

• No “touch” or “release” – Detecting user intention more difficult • No physical boundary

• “Live” interaction vs post-processed result

�Optical 3D gestures can complement existing systems• Off screen/over-screen sensing volume

• New uses cases • Wakeup or UI response as user approaches• Hands-free interaction (many ideas)• Gaming controller

25

1D Gesture Detection 26

• Time domain• Distance

• Amplitude• Signal & Noise

• Time of Flight IR2 Outputs

• Distance

• Reflectance• Surface

• % fill factor

• Multiple objects

2 Unknowns2 Unknowns• Object Properties

• Distance measurement alone

• Tap / double-tap

• Up/down level control

• IF we assume fixed object reflectance

• Lateral motion can be estimated from a single pixel!

Lateral Motion Estimation

• Assume

• Surface properties are stable (same object) within a given time period

• � only change must be due to % filled FOV (represents x-y motion)

• We can therefore calculate the % of the Field of View filled by the object

• Independent of object distance!

27

20% 80% 100% 80% 20%

Amplitude can be normalized (using distance info)

% Field-of-View Coverage

• Object reflectance modeled at all distances

• Model needs to include non-linearities

28

Sensor saturation

Emitter blocked

3D Gesture Detection 2 ToF Pixels 29

• Spatially or angularly separated ToF detectors

• Linear continuous slider

• Smooth triangulation of position/speed in X (horizontal) and Z (vertical)

Gesture Definitions 4 ToF Pixels

• Continuous control � hover & tilt

• Hover & tilt

• Great gameplay

• Can act as mouse/track pad

• Post-processed movement examples

30

1 2

1

2

Swipe Press Wave

More Gesture Definitions

• Movement properties that can be detected

• Motion lateral/angular speed

• Object width

• Closed vs spread fingers

• Hand-tilt detection

• Goal is for ROBUST detection of gestures/motion

31

Flat hand swipe

4 fingers swipe

Tilted hand swipe

Tilted hand swipe

What’s Next?

• 3D Gestures is a wide field � more to come!

32

33

Q&A