3d light-field technology in combination with deep-learning for...

3D Light-field Technology in Combination with Deep-Learning for Industry

Philipp Schmid – Head Robotics & Automation

Copyright 2019 CSEM | Page 1

Our mission

Development and transfer of microtechnologies to the industrial sector – in Switzerland, as a priority – in order to

reinforce its competitive advantage.

Cooperation agreements with established companies

Encouraging the creation of start-ups

Copyright 2018 CSEM | Big Data – Smart Data – Fachgruppe Smart Systems; 12. Oktober 2018 | Alexander Steinecker| Page 2

CSEM – A TECHNOLOGY ORGANIZATION IN SWITZERLAND

The CSEM is a Swiss RTO with over 450 emps. Thecompany develops advanced microtechnologiesfor the industry. Thus it creates innovations.Focus areas are Indusry 4.0, e-health, advancedmanufacturing, clean energy.CSEM’s head quarters are located in Neuchâtel,Further branches are in Muttenz, Landquart,Alpnach and Zürich. CSEM is connected toInternational RTO’s such as FhG, VTT, CEA.In the past 10 years CSEM created over 40 Startup companies active in numerous markets.

Copyright 2018 CSEM | Big Data – Smart Data – Fachgruppe Smart Systems; 12. Oktober 2018 | Alexander Steinecker| Page 3

CSEM at a glance

437Persons

43New ventures

175Industrialclients

203Patent families

64Europeanprojects

43Nationalities

83.0Turnover(mio CHF)

Copyright 2019 CSEM | Page 4

CSEM – Swiss Research for Industry

N N A L M Z

N A L

MZ

N

CSEM Zürich

CSEM Muttenz

CSEM Neuchâtel

CSEM Alpnach

CSEM Landquart

Copyright 2019 CSEM | Page 5

Technology platforms to foster innovation

Advanced micro-manufacturing

Printableelectronics

Bio-surface engineering

MEMS

Nano-surface engineering

Scientific instrumentation

Medical technologies

Automation

Energy systems

Emerging & thin-film PV

PV cells& Modules

Vision systems

System-on-chip

Wireless

Industry 4.0

IoT

e-health

e-energy

Advanced manufacturing

Functionalpackaging

Copyright 2019 CSEM | Page 6

Robotics & Automation

Robotics

Vision

Control

Deep Learning

Copyright 2019 CSEM | Page 8

History

1908Gabriel Lippmann

Copyright 2019 CSEM | Page 9

100 years later…

2008Gabriel Lippmann

Market entrance 2010

Light field cameras for industry and research

Patented MLA design for optimal depth-of-field

Copyright 2019 CSEM | Page 10

Principle

Standard Camera Lightfield Camera

Copyright 2019 CSEM | Page 11

Lightfield (Plenoptic) Camera – 3D Vision

GPU Power

Copyright 2019 CSEM | Page 12

3D Lightfield camera

Copyright 2019 CSEM | Page 13

3D Lightfield camera

Copyright 2019 CSEM | Page 14

Extended depth-of-field

Lens Type 1 Lens Type 2 Lens Type 3

Copyright 2019 CSEM | Page 15

Multi Focus

Image plane Microlens plane

Lens Type 1

Lens Type 2

Lens Type 3

Copyright 2019 CSEM | Page 16

Extended Depth-of-field

Raytrix Lightfield Camera R5μStandard 4MP Camera

Copyright 2019 CSEM | Page 17

Occlusion

Stereo Camera Lightfield Camera

Copyright 2019 CSEM | Page 18

3D in production

Total Focus Image

3D imageRoundness

Statistics

Copyright 2019 CSEM | Page 19

Difficult Parts

Copyright 2019 CSEM | Page 20

Screws

Copyright 2019 CSEM | Page 21

BinPicking for small parts (under development)

• 3D Lightfield Camera

• CSEM Libraries

• Fast and Robust

Copyright 2019 CSEM | Page 22

Dental

Copyright 2019 CSEM | Page 23

3D Microscopy

FoV ~ 1x1mm Total Focus 3D

Copyright 2019 CSEM | Page 24

Solder Bumps

Copyright 2019 CSEM | Page 25

Product range

Copyright 2019 CSEM | Page 26

Quality Control Today

Copyright 2019 CSEM | Page 27

Traditional Image Processing…

Model

World

≠

Algorithm

Copyright 2019 CSEM | Page 28

Copyright 2019 CSEM | Page 29

How does our Brain work?

DEEP LEARNINGAT CSEM

10+ years experience

Support humans

Industry proven

Copyright 2019 CSEM | Page 31

Deep Learning since 2008 => Industrially validated solutions

CSEM

technology

transfer

Copyright 2019 CSEM | Page 32

Copyright 2019 CSEM | Page 33

Missing Link

Copyright 2019 CSEM | Page 34

Integrated Robotic Software Framework «VISARD»

PC-based, one stop solution

Flexible, extendable,

reusable and user-friendly

Database logging and

automatic management

report generation

Hardware independent

Online process

development with scripting

tool (similar IEC 61131)

Benefits for the users

Copyright 2019 CSEM | Page 35

Resolution

Parameters Overview system High-resolution system

Working distance 48 mm 20 mm

Field of view 7.8 mm x 7.8 mm 0.65 mm x 0.44 mm

Lateral resolution 3 m 0.7 m

Depth of field 1.1 mm 0.06 mm

Depth resolution 2.5 m 0.4 m

Framerate (max) 81 fps 30 fps

www.citcom.eu

Funded by the European Union’s HORIZON 2020 programme, Grant Agreement no. 768883

Copyright 2019 CSEM | Page 36

3D Vision Demo

Copyright 2019 CSEM | Page 37

CSEM Vision Benefits

• CSEM source code – no third party libraries (Cognex, Halcon, Matrox)

• No third party license fees

• No black box, not depended on external support

• CSEM has control over every bit and byte -> perfect adjusted solution

• Use of latest machine learning technics

• Modular software in C# .NET (supporting many camera brands)

• Easy to customize

• Full data logging (folder structured or database)

• Connection to PLC (ADS, OPC, …), TCP/IP, RS232, …

Contact:

Philipp Schmid

Head Robotics & Automation

psd@csem.ch

T +41 41 672 75 62

M +41 78 769 86 78

CSEM SA

Untere Gründlistr. 1 | 6055 Alpnach Dorf

www.csem.ch