In-sensor processing for high-speed low-power 2D/3D imaging

Ricardo Carmona-Galán, PhDInstituto de Microelectrónica de Sevilla IMSE-CNMCSIC-Universidad de Sevilla, Spain

rcarmona@imse-cnm.csic.eshttp://www.imse-cnm.csic.es/~rcarmona

Low-power electronics

Mixed-signal processing

Systems-on-Chip

Deep submicron CMOS

Hybrid IC technologies

Smart sensory systems

Biomedical signal

monitoring

High-speed and secure

comm.

Bioinspired signal

processing

• 7 Labs with scientific instrumentation for logical, electrical, functional and thermal characterization of mixed signal, RF and optoelectronic ICs:

~150 prototypes in 20 years

• Anechoic chamber

• Pulsed laser

• Industrial test station

• Probe station

52%

14%

27%7%

Scientific staff

Technical support

PhD Students

Administration0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

Male/Female Perm/Temp CSIC/U.Seville

74

4940

26

5160

• Low-power and low-voltage analogue and mixed-signal design

• Analogue-to-digital converters and mixed-signal interfaces

• Implantable and wearable intelligent biosensors

• Smart image sensors and vision chips

• Heterogeneous processing systems and 3D-integrated circuits

Research Group on Integrated Interface Circuits and Sensory Systems

© afran.org.au

© ESA 8

© Satoshi Hasegawa

• High resolution images → lots of data and fast readout

• Low-noise detection

• Low power consumption

• Embedded functionalities?

10

(Image) (Image, signal, flag, etc.)

Energy:

Time:

𝐸tot = 𝑁op ∙ 𝑒0

𝑇tot =𝑁op

𝑁proc𝑡0

Time-critical applications

Throughput =𝑁op

𝑇tot=𝑁proc

𝑡0

Power-aware applications

Power =𝐸tot𝑇tot

= 𝑁proc ∙𝑒0𝑡0

Power and time-critical applications

© Paula Henihan

FOM =Throughput

Power=

1

𝑒0

Input𝑁op

Output

11

𝒆𝟎 =𝐧𝐉

𝐨𝐩.=𝐧𝐖

𝐨𝐩𝐬=

𝟏

𝐌𝐎𝐏𝐒/𝐦𝐖=

𝟏

𝐆𝐎𝐏𝐒/𝐖

12

13

@ circuit level

→ Power efficient circuits

→ High signal/bias ratio

→ Complex dynamics

@ architectural level

→ Distributed processing

→ Distributed memory

→ Distributed ADC

Minimization of e0

Imager

Our hypothesis → the closer to the sensor the faster and the more power efficient in extracting the relevant data

• Data transfer bottlenecks• Large memory footprint• High computational demand

• Multiple scales required• Many feature maps in parallel• Lots of irrelevant data

ROI of our research

14

Feature extractor

Classifier

[M. Suárez et al. IEEE JSSC 2017]

Scale-spaceextrema

detection

Keypointlocalization

Orientation assignment

Keypointdescriptor

SIFT: Scale Invariant Feature Transform

The first stage of computation searches over all scales

and image locations

[D. Lowe, IJCV 2004]16

3% 5%

92%

Distribution of the computational effort in SIFT

Key point extraction

Descriptor vector

Gaussian pyramid

Source: K. Mizuno et al., “Fast and Low-Memory Bandwidth Architecture of SIFT Descriptor Generationwith Scalability on Speed and Accuracy for VGA Video” FPL 2010

HDTV (1920 x 1080 px)

@30 fps

3375 #keypoints

17

Focal-plane low-power feature extractor chip

[J. Fernández-Berni et al. IEEE JSSC 2011]

34mm

28mm

18

[J. Fernández-Berni et al. IEEE JSSC 2011]

Technology: 0.35μm CMOS 2P4M 3.3V

Die size (w/pads): 7280.8μm × 5780.8μm

Cell size: 34.07μm × 29.13μm

Fill factor: 6.45%

Resolution: QCIF (176×144 px)

Photodiode type: n-well/p-substrate

FPN: 0.72%

PRNU (50% SR): 2.42%

Sensitivity: 0.15V/(lux·s)

Power cons.: 5.6mW@12kSa/s

On-chip scale-space generation Edge detectionid

eal

chip

RMSE = 24.99% RMSE = 19.39% RMSE = 6.17% RMSE = 3.58% RMSE = 6.68%

t = 40ns t = 100ns t = 400ns t = 800ns t = 1500ns

idea

l

chip

inp

ut

Energy-based representation

20

Haar-like features detection

[Viola & Jones, CVPR 2001]

http://makematics.com/research/viola-jones/

21

Integral image and II2 extractor chip19.6mm

17mm

[J. Fernández-Berni et al. IJCTA, 2015]22

Technology CMOS 0.18mm 1.8V 1P6MDie size (with pads) 7.5mm × 5mmPixel size 19.59mm × 17mmImage resolution 320 × 240 pixelsFill factor 5.4%Photodiode type n-well/p-substratePower supply 3.3 (pads), 1.8V (core)DSNU/PRNU (50% SR) 1.7% / 3.5%ADC throughput 5MSa/s (200ns/Sa)

Power consumption @30fps (mW)Image capture 42.6+ Programmable pixelation 43.8+ Block-wise HDR 42.6+ Integral image 55.2+ Gaussian filtering 43.1

[J. Fernández-Berni et al. IJCTA, 2015]

23

Prototype output iPad output

[J. Fernández-Berni et al. TCAS-II, 2016]

Image pixels histogram

Major drawbacks

Reduced fill factor

Large pixel pitch

→ Limited sensitivity

→ Small image size

→ Spatial aliasing

Major achievements

Concept demonstration

Programmable embedded functionalities

Image-to-Decision chain at >1,000fps using 60nW per pixel (industrial chip)

Spatial Gaussian filtering @20nJ/filter

Content-aware HDR acquisition with >145dB intra-frame DR

25

Samsung’s Newest ISOCELL Image Sensor Enables Mobile Devices to ‘Slow Down’ TimeBussinessWire: February 26, 2018

“Samsung introduces the 3-stack ISOCELL Fast 2L3. The 1.4-μm 12MP image sensor with 2Gb of integratedLPDDR4 DRAM delivers fast data readout speeds for super-slow motionand sharper still photographs with lessnoise and distortion.”

2Gb LPDDR4 DRAM layer

1.4-μm 12MP image sensor layer

Analog and logic layer

26

44um

mm

mm

44u

m

22um

[M. Suárez et al. IEEE JSSC 2017]27

HW Solution Frame size Energy/frame Energy/px Mpx/s

GP chip180nm CMOS

176 x 120 px 70mW @ 8ms0.56mJ/frame

26.5nJ/px 2.64

Ref. [1] OV9655 + Core-i7

640 x 480 px 90mW + 35 W @ 136ms4.8J/frame

15.5uJ/px 2.26

Ref. [2] OV9655+ Core-2-Duo

640 x 480 px 90mW + 35 W @ 2.1s73.7J/frame

240uJ/px 0.15

Ref. [3] OV9622 +Snapdragon S4

350 x 256 px 30mW + 4 W @ 98.5ms0.4J/frame

4.4uJ/px 0.91

[1] M. Murphy et al. “Image Feature Extraction for Mobile Processors”, IEEE IIWSC 2009

[2] Feng-Cheng Huang et al. “High-Performance SIFT Hardware Acceleration for Real-Time Image FeatureExtraction”, IEEE TCAS-VT, Vol. 22, No. 2, March 2012

[3] G. Wang et al. “Workload Analysis and Efficient OpenCL-based Implementation of SIFT Algorithm on aSmartphone”, IEEE GlobalSIP 2013

Comparison of Gaussian pyramid implementations

28

Features from compressed samples

0 500 1000 1500 2000 2500 3000 3500 40000

2

4

6

8

10

12

14

16

18

20

Number of SamplesA

vera

ge D

ista

nce

Harris-Nesta

Nesta

[M. Trevisi et al., ICECS 2015]

Original 64x64-pixel image Harris-corners on original

Harris on reconstructed Harris on compressed samples

29

Time-encoding pixel

Technology CMOS 0.18mm 1P6M

Die size (w. pads) 3174μm × 2227μm

Pixel size 22μm × 22μm

Fill factor 9.2%

Image resolution 64 × 64 pixels

Photodiode type n-well/p-substrate

Power supply 3.3V-1.8V

Predicted power consumption <100mW

Frame rate 30fps

Max. compressed sampling rate 50kHz

Clock Freq. 24MHz

[M. Trevisi et al., DATE 2018] 31

Asynchronous self-reset pixels

[J. A. Leñero et al. IEEE JSSC, 2017]

Wide linear dynamic range by asynchronous self-reset and tagging of saturation events [J. A. Leñero et al. IEEE JSSC, 2017]

Wide linear dynamic range by asynchronous self-reset and tagging of saturation events [J. A. Leñero et al. IEEE JSSC, 2017]

Linear HDR CMOS imager + NIR lens LWIR camera (FLIR Lepton)

[J. A. Leñero et al. IEEE Sensors J, 2018]

36

Sun Sensor based on a Luminance Spiking Pixel Array

DR = 70dB

37[J. A. Leñero et al. IEEE Sensors J, 2017]

Fig. from Prof. Dr. Johanna Stachel’s talk “Detectors in Nuclear and Particle Physics”. Dep. Physics and Astronomy, Univ. of Heidelberg, July 2014 38

Fig. from Beyer T, Townsend DW, Czernin J, Freudenberg LS. “The future of hybrid imaging—part 2: PET/CT”. Insights into Imaging. 2011;2(3):225-234. doi:10.1007/s13244-011-0069-4. 39

ToF in CMOS technology

MethodIndirect ToF

Direct ToFPM CW

Charge integration

Single-photon detection

Photoncounting

Time of arrival

40

Photo from Zappa et al. Sens. & Act. A:Phys. 2007

DV

DI

2

3 11

SPAD: single-photon avalanche diode

2

31

VD

ID

I D (

mA

)

time (s)

41

PLL

Analog I/O buffers and voltage reference

64×64 3D imager

Data serialiser

Co

ntr

ol

sig

nal

s d

istr

ibu

tio

n t

ree

ST

AR

T/S

TO

P d

istr

ibu

tio

n t

ree

Ro

w d

eco

der

5 mm

5 m

m

64 µm

64

µm

TDC

AQR

Mem

ory

& b

uff

ers

SPAD

12

µm

[I. Vornicu et al. Proc. SPIE, 2014]

64× 64 CMOS SPAD Imager with in-pixel TDC

42

Active quenching/recharge circuit with time-gatingVspad+

M1

M2

M3

M6

Inv1

VDD

K

A

Vout

Vrestore

VDD

Vhold_off

Inv2

Inv3

M7

M8

M9M10

Vsense Vcap

M4

M5 Vgate

M3: active quenchingM6: active rechargingM4,5: time-gating

P+/ N-wellSPAD

Adjustable

time-constant

By gating the front-end of the SPAD, the power drained from VSPAD+ drops from 330mW to less than 10mW

[Vornicu et al. ISCAS, 2013]

43

Interpolative TDC based on a VCRO

Ripple

counter

Encoder

R

Ext_Stop

TUNE RO

8b coarse

[B10:B3]

3b fine [B2:B0] <10:0>

Final state

MUX

2D/ 3D

Vout

To in-pixel

memory

out1Start/Stop

logic

Ext_StartVout

Sel_Start

EN_TDC

• Ripple counter → encode M events with the m coarser bits B10:B3

• 2n VCRO phases → encode N events with the n finer bits B2:B0

𝑇𝑚𝑒𝑎𝑠 = 𝑀 +𝑁

2𝑛𝑇𝑜𝑠𝑐 𝑇𝐿𝑆𝐵 =

𝑇𝑜𝑠𝑐2𝑛

8 phases time interpolation

fCLK ~ 860Mhz

TLSB=145ps

fCLK ~ 7Ghzfor a simple

ripple counter

44

45

• Tolerance to magnetic fields

• Compactness

• Low bias voltage

• Better spatial resolution

• High gain (105-106)

• Low power

• Cheaper in production

• Focal-plane image processing???

SiPMs vs. PMTs in PET

46

Inaccurate determination of DOI results in incorrectly positioned LOR

Incorrectly positioned LOR leads to parallax errors and, hence imprecise reconstruction of the image

[Kao et al. 2000]

[Hoffman et al. 1989]

Depth of interaction (DOI) estimationƔ-ray High-energy

photons

Scintillator

crystal

DOI

Visible

photons

32µm 32µm

Imager

Back-end

circuits

TDC,

logic control &

memory

Scintillators

array

47[Lerche et al. 2005]

Depth of interaction (DOI) in continuous scintillator crystals can be inferred from width of light-distribution

200 zz

Vspad+K

A

AQR

GNDspad+

Vspad+K

A

AQR

GNDspad+

Vspad+K

A

AQR

GNDspad+

Vspad+K

A

AQR

GNDspad+

VD

DV

DD

EN

RS

T

EN

RS

T

ENRST

ENRST

ENRST

VDD VDD

OVF

B12

B1

OVFB12

B1

OV

F

B1

2

B1

OV

F

B1

2

B1

VD

D

OVF

CR1

VR1

CRN

VRN

CC1

VC1

CCM

VCM

PCM

PRN

PC1

PR1

NR NR

NC NC

NR NR

NC NC

P(N,1) P(N,M)

P(M,1)P(1,1)

CMOS technology UMC 0.18um

No. of SPADs 8 x 8

Die area 1.5 x 1.5 mm2

Sensor area 256 x 256 um2

SPAD cell pitch 32 x 32 um2

SPAD diameter 12um (14um TWELL)

Counters depth 13b

Time resolution <165ps

8 x 8 SPAD array for DOI estimation

Ro

w c

ou

nter

Col decoder

Ro

w d

eco

der

Col countersData serialiser

Dat

a se

rial

ise

r

8x8 SPADs +AQR

Ro

w c

ou

nter

Col decoder

Ro

w d

eco

der

Col counters

Data serialiser

Dat

a se

rial

ise

r

8x8 SPADs + AQR

[Vornicu, IEEE Sensor J, 2016] 48

32um

32

um

SPAD

Active quenching and reset

Rea

do

ut

Pulsed laser

Lenses

SPAD chip

Beam splitter

0 5 10 15 20 25 30 35 40

200

400

600

800

1000

1200

1400

Time [ns]

Co

un

ts

VSPAD+

=10.9V

DT=15nsDCR=3.7Khz

=447nmf

laser=20Mhz

FWHMLASER

=68ps

Poptical,avg

=18nW

Measured overall FWHM = 180ps

Laser FWHM = 68ps

Synchronization jitter = 20ps

Estimated time resolution < 165ps

R1|R

2|R

3|R

4|R

5|R

6|R

7|R

8

R1|R

2|R

3|R

4|R

5|R

6|R

7|R

8

C1|C2|C3|C4|C5|C6|C7|C8C1|C2|C3|C4|C5|C6|C7|C8

1 2 3 4 5 6 7 8200

400

600

800

1000

1200

1400

1600

1800

Row/Column index

Dark

counts

R1-R8

C1-C8

1 2 3 4 5 6 7 8

1000

2000

3000

4000

5000

6000

Row/Column index

Tota

l counts

R1-R8

C1-C8

DCR profile Light spot profile49

Scintillatorcrystal array

Hybrid semiconductordetector array

Photomultiplier tubesor Si-PMs

Indirect detection

Direct detection

Scintillatorcrystal array

Si-PMs

Pulse and coincidence processing

Pulse and coincidence processing

Pulse and coincidence processing

CMOS-enabled additions

- Pulse discrimination- Pulse shape analysis- Timing of the pulses- Coincidence window

evaluation

- Single-photon detection- Pulse shaping- Photon energy

discrimination- Dead-time control

- Multispectral imaging- Combined PET/CT or PET/MR- Single-chip 3D integration

Integrated electronics in PET detectors

50

CSIC-Universidad de Sevilla

Jorge Fernández BerniÁngel Rodríguez Vázquez Rocío del Río FernándezJuan Antonio Leñero BardalloIon VornicuMarco Trevisi

Universidade de Santiago de Compostela

Manuel Suárez CambreVíctor Brea Sánchez Diego Cabello FerrerPaula López Martínez

Universidade Federal do Rio de Janeiro

Fernanda D. V. R. OliveiraJosé Gabriel R. C. Gomes