isaac an accelerated robotics framework hammad mazhar€¦ · hammad mazhar. 2 credit: yevgen...
TRANSCRIPT
![Page 1: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/1.jpg)
ISAACAn Accelerated Robotics Framework
Hammad Mazhar
![Page 2: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/2.jpg)
2
Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine
PR2 - HOCKEY
![Page 3: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/3.jpg)
3
PROJECT ISAAC: HOCKEY
Teach a robot how to play hockey using ground truth
Robot: PR2 (Willow Garage)
Kinematic model
7 joints, 1 arm
Observations:
Puck and Joint positions, velocities
Goal location
3
![Page 4: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/4.jpg)
4
PROJECT ISAAC
![Page 5: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/5.jpg)
5
DISCO HOCKEY BOT
5
![Page 6: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/6.jpg)
6
ISAAC LAB: DOMINOES
6
![Page 7: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/7.jpg)
7
SYNTHETIC DATA GENERATION• Generate 50K images
• Multiple distractors, lights, etc
• Labeled:
• Location, pose, pip count
7
![Page 8: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/8.jpg)
8
GAMEPLAY LOGIC TRAINING (SLOW MOTION)
8
![Page 9: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/9.jpg)
9
ISAAC PLAYS DOMINOES
![Page 10: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/10.jpg)
10
ROBOTICS AT NVIDIATransfer learning in Isaac Lab
![Page 11: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/11.jpg)
11
ISAAC PLAYS GOLF
![Page 12: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/12.jpg)
12
VIRTUAL DEMONSTRATION FOR ROBOTICS
![Page 13: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/13.jpg)
13ISAAC SDK
![Page 14: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/14.jpg)
14
ISAACSDK FOR ROBOTICS
Simulation EnginePhoto-realistic Graphics ∙ Physics ∙
∙ Procedural Generation ∙ Massive parallelism
∙ Unreal Engine 4
Isaac FrameworkCodelets ∙ Behaviors ∙ 3D Poses ∙ Distributed
∙ Messaging ∙ Synchronization ∙ Record & Replay
∙ Configuration ∙ Visualization
World modelWarehouse ∙ Office
∙ Store ∙ Home
Robot modelCarter ∙ URDF loader
Virtual Sensors Virtual Actuators Sensor Processing Actuator Control HW Sensor HW Actuator
MLTensorRT ∙ CUDA
∙ Tensorflow ∙ ...
GemsOptimizers ∙ Algebra
∙ EKFs ∙ Depth ∙ ...N
avig
atio
n
Be
ha
vio
rs
Inte
ractio
ns
with
hu
ma
ns
Manip
ula
tion
Pe
rce
ptio
n
DriversLidar ∙ Camera ∙ IMU ∙
Robot Base ∙ ...
Unified Message APIUse the same messages for simulation,
actual hardware and across all apps
Simulate DeployDevelop
JetsonFully integrated with
X2 and Xavier
![Page 15: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/15.jpg)
15
ISAAC SDK: A ROBOTICS TOOLBOX
Jetson AGX XAVIER
Advanced GPU, 64-bit CPU, Video CODEC, VIC, ISP
CUDA, Linux4Tegra
Core libraries Navigation Reference DNN
VisionWorks, OpenCV, TensorRT, cuDNN
ToolsDrivers
![Page 16: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/16.jpg)
16
NODE BASED PROGRAMMING INTERFACE
![Page 17: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/17.jpg)
17
Global planner
NAVIGATION GEMS
Global planner Obstacle avoidance Localization State machine
![Page 18: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/18.jpg)
18
STEREO DNN
![Page 19: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/19.jpg)
19
DEVELOPER TOOLS
![Page 20: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/20.jpg)
20
ISAAC SIMULATION
![Page 21: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/21.jpg)
21
SENSOR SIMULATIONCameras and Lidar
![Page 22: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/22.jpg)
22
MANIPULATIONURDF Support
![Page 23: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/23.jpg)
23
ROSInteroperability with existing robotic stacks
![Page 24: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/24.jpg)
24
![Page 25: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/25.jpg)
25
DESIGNING MEANINGFUL TRAINING ENVIRONMENTS
25
![Page 26: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/26.jpg)
26
DESIGNING MEANINGFUL TRAINING ENVIRONMENTS
26
![Page 27: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/27.jpg)
27
PHYSX 4.0
![Page 28: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/28.jpg)
28
CARTER
![Page 29: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/29.jpg)
29
CARTER REFERENCE DESIGN
Velodyne VLP16 Zed stereo IMUSegway
Carter
Jetson AGX Xavier
![Page 30: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/30.jpg)
30
Iterate at high speed on a desktop and in simulation
Deploy and test in reality
Training on DGX
CARTER REFERENCE DESIGN
Xavier and TX2
Carter app
![Page 31: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/31.jpg)
31
CARTER WITH ISAAC SDK
![Page 32: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/32.jpg)
32
SALAD DELIVERY
![Page 33: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/33.jpg)
33
NEXT STEPS
![Page 34: ISAAC An Accelerated Robotics Framework Hammad Mazhar€¦ · Hammad Mazhar. 2 Credit: Yevgen Chebotar, Karol Hausman, Marvin Zhang, Sergey Levine PR2 - HOCKEY. 3 PROJECT ISAAC: HOCKEY](https://reader033.vdocument.in/reader033/viewer/2022043006/5f903cec2773b447d4047ba2/html5/thumbnails/34.jpg)
34
July 2018 2019 2020
CARTER IS JUST THE BEGINNING