robots that see like humans
Post on 12-Apr-2017
561 Views
Preview:
TRANSCRIPT
Robots that see likeRobots that see likehumanshumans
Real Time Stereo Vision With ZED Camera on Nvidia Jetson TX1
Walter Lucettimyzhar@robot-home.it
Summary
21/05/2016Officine Robotiche 2016
2
Walt
er
L uce
tti –
myzh
ar@
rob
ot-
hom
e.it
Live Demo
Conclusions
Introduction
Stereo “Binocular” Vision
Robotic Stereo Vision
“Simplified” Stereo Vision
Pin Hole Camera model
Stereo Vision camera model
Stereo Vision process
A Robotic Stereo Vision System
Nvidia Jetson TX1
Stereolabs ZED camera
Stereolabs ZED SDK
Stereolabs ZED engine
Simple Code example
Introduction3
Introduction4
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Can a robot replicate the complicatedhuman binocular vision system?
21/05/2016Officine Robotiche 2016
Stereo “Binocular” Vision 5
[from Wikipedia: Binocular vision]
Vision made with TWO “EYES”
It gives a “creature” a spare eye in case one is damaged
It gives a wider field of view
It can give stereopsis in which binocular disparity (or parallax) provided by the two eyes' different positions on the head gives precise depth perceptionIt allows a creature to see more of, or all of, an object behind an obstacle
It gives binocular summation in which the ability to detect faint objects is enhancedW
alte
r L u
cett
i – m
yzh
ar@
rob
ot -
hom
e.it
21/05/2016Officine Robotiche 2016
Robotic Stereo Vision6
How to replicate Human Stereo Vision? Two synchronized camera sensors
A solid mechanical structure
A powerful elaboration system
and if we want to be more precise, but we want to complicate the system:
Two motors to replicate vergence capability
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
21/05/2016Officine Robotiche 2016
“Simplified” Stereo Vision7
“Simplified” Stereo Vision8
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
A complete Stereo Vision system has vergence capability, but is too complicated
We will study a Simplified Stereo Vision system where cameras cannot rotate
21/05/2016Officine Robotiche 2016
9
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Pin hole camera model
Camera Plane
Optical Axis
Image coordinatesWorld coordinates
World to image
Image to world
Without “Z” we cannot estimate the real dimension of an object using a single camera!
“f” can be estimatedcalibrating thecamera system
This is a simplifiedcamera model thatdoes not take care
of Optical Distorsion
andSensor
Imperfections
21/05/2016Officine Robotiche 2016
u=f ∗x
z
v=f ∗y
z
x=u∗z
f
y=v∗z
f
10
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereo Vision model
Very important parameter:● Baseline: b
From triangles similarity:
Very important value:● Disparity: D = u
l - u
r
From disparity we can calculate howmuch a point is far from a camera:THE “Z” VALUE!!!
21/05/2016Officine Robotiche 2016
fZ P
=ul
X P
fZ P
=−ur
b−X P
Z P=b∗ f
u l−ur
Calculate “Disparity” (D) for each point of the image
11
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereo Vision process
Calculate the depth (Z) ofeach point of the image
Calculate X and Y of eachpoint of the image
Algorithm
MATCHING
REQUIRES A LOT OF
COMPUTATIONALPOWER
D = ul - u
r
21/05/2016Officine Robotiche 2016
12
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereo Vision summary
Allows to calculate real world coordinate of an objectRequires a solid mechanical structureRequires a precise calibration of the optical systemRequires a lot of computational power…Is more complicated than what seen until now
21/05/2016Officine Robotiche 2016
A Robotic Stereo Vision system13
14
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
A Robotic Stereo Vision System
Nvidia Jetson TX1 www.nvidia.com/object/jetson-embedded-
systems.html
Stereolabs ZEDwww.stereolabs.com
21/05/2016Officine Robotiche 2016
15
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Nvidia Jetson TK1
Main features
Nvidia Tegra X1 SOC
NVIDIA Maxwell GPU with 256 CUDA cores
NVIDIA 4-Plus-1™ 2.3 Ghz quad-core ARM® Cortex™-A15 CPU
4 GB LPDDR4 Memory
16 GB eMMC 5.1 Flash Storage
1 full-size SD/MMC connector
1 full-size HDMI port
1 USB 2.0 port, micro AB
1 USB 3.0 port, A
1 RTL8111GS Realtek GigE LAN
GPIO, SPI, I2C, UART, CAN
WiFi & Bluetooth connection
...
Native OpenCV support with GPU module for Real Time computer vision
ROS enabled
Ubuntu powered
and more...
Android ...
21/05/2016Officine Robotiche 2016
16
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereolabs ZED cameraMain features
Auto calibrating and auto sync
USB 3.0
High Res at high FPS2208x1242 15 Hz
1920x1080 30 Hz
1280x720 60 Hz
640x480 120 Hz
Simple and well documented C++ SDK
Automatic Depth Map generation
3D Range: 1 m to 20 m(12 cm baseline)
Outdoor and indoor
Jetson TX1 driver
ROS driver
OpenCV support
21/05/2016Officine Robotiche 2016
17
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereolabs ZED SDK
C++ Classes and APIs for Linux and Windows
2 main classes and 5 support classes
Main classes:
Camera: the main class to use the ZED camera
Mat: the class to store image data in GPU and CPU memory spaces
Support classes:
CamParameters: calibration params for the two sensors
StereoParameters: calibration params for stereo processing
resolution: dimensions of the images
float3Struct: 3D point
uchar3Struct: RGB pixel
Current version:
0.9.3-beta for Linux and Windows, 0.9.3-alpha for Jetson TX1
21/05/2016Officine Robotiche 2016
18
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereolabs ZED engine
Stereo results are availablein GPU memory
Direct processing by CUDAor download to CPU memoryfor CPU processing
21/05/2016Officine Robotiche 2016
19
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Stereolabs ZED tools
ZED Explorer ZED Depth Viewer
21/05/2016Officine Robotiche 2016
20
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Simple code example
//ZED Includes#include <zed/Camera.hpp>
int main(int argc, char **argv){ sl::zed::SENSING_MODE dm_type = sl::zed::RAW; sl::zed::Camera* zed;
// Camera object creation zed = new sl::zed::Camera(sl::zed::HD2K); // We can use different resolutions: // HD2K HD1080 HD720 VGA
// Sizes of the images int width = zed->getImageSize().width; int height = zed->getImageSize().height;
// Initialization of the camera sl::zed::ERRCODE err = zed->init( sl::zed::MODE::PERFORMANCE );
if (err != sl::zed::SUCCESS) { delete zed; return 1; }
// Stereo processing zed->grab(dm_type);
// CPU Data sl::zed::Mat depth_cpu; sl::zed::Mat left_cpu; sl::zed::Mat right_cpu;
// Data acquiring on CPU memory depth_cpu = zed->retrieveMeasure(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage(sl::SIDE::LEFT); right_cpu = zed->retrieveImage(sl::SIDE::RIGHT);
// GPU Data sl::zed::Mat depth_gpu; sl::zed::Mat left_gpu; sl::zed::Mat right_gpu;
// Data acquiring on GPU memory depth_cpu = zed->retrieveMeasure_gpu(sl::MEASURE::DEPTH); left_cpu = zed->retrieveImage_gpu(sl::SIDE::LEFT); right_cpu = zed->retrieveImage_gpu(sl::SIDE::RIGHT);
return 0;}
21/05/2016Officine Robotiche 2016
Live demo21
Conclusions22
23
Wal
ter
L uce
tti –
myzh
ar@
rob
ot -
hom
e.it
Conclusions
With Stereo VisionA robot can see obstaclesA robot can measure obstaclesA robot can localize obstacles
Stereo Vision requiresa lot of mathematical elaborations a lot of computational powera solid mechanical structure
so
Nvidia Jetson TX1 with Stereolabs ZED is a really amazing solution for Robotic Stereo Vision!
21/05/2016Officine Robotiche 2016
The End!
OR
THANK YOU FOR YOUR ATTENTION!
Walter Lucettihttp://myzharbot.robot-home.it
GitHub@myzhar
YoutubeMyzhar
Twitter@myzhar
Facebookrobothome
24
21/05/2016Officine Robotiche 2016
top related