what is pytorch?on-demand.gputechconf.com/gtc/2018/presentation/s... ·...
TRANSCRIPT
![Page 1: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/1.jpg)
![Page 2: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/2.jpg)
Adam Paszke, Sam Gross, Soumith Chintala, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Alban Desmaison, Andreas Kopf, Edward Yang, Zach Devito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy & Team
![Page 3: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/3.jpg)
What is PyTorch?
![Page 4: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/4.jpg)
What is PyTorch?
Ndarray library with GPU support
automatic differentiation engine
gradient based optimization package
Deep Learning
Reinforcement LearningNumpy-alternative
Utilities (data loading, etc.)
![Page 5: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/5.jpg)
ndarray library•np.ndarray <-> torch.Tensor •200+ operations, similar to numpy •very fast acceleration on NVIDIA GPUs
![Page 6: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/6.jpg)
ndarray library
Numpy PyTorch
![Page 7: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/7.jpg)
ndarray / Tensor library
![Page 8: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/8.jpg)
ndarray / Tensor library
![Page 9: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/9.jpg)
ndarray / Tensor library
![Page 10: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/10.jpg)
ndarray / Tensor library
![Page 11: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/11.jpg)
NumPy bridge
![Page 12: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/12.jpg)
NumPy bridge
Zero memory-copy very efficient
![Page 13: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/13.jpg)
NumPy bridge
![Page 14: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/14.jpg)
NumPy bridge
![Page 15: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/15.jpg)
Seamless GPU Tensors
![Page 16: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/16.jpg)
automatic differentiation enginefor deep learning and reinforcement learning
![Page 17: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/17.jpg)
PyTorch Autogradfrom torch.autograd import Variable
![Page 18: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/18.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
PyTorch Autograd
![Page 19: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/19.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t())
MMMM
PyTorch Autograd
![Page 20: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/20.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h
MMMM
PyTorch Autograd
![Page 21: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/21.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h
Add
MMMM
PyTorch Autograd
![Page 22: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/22.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh()
Add
MMMM
Tanh
PyTorch Autograd
![Page 23: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/23.jpg)
from torch.autograd import Variable
x = Variable(torch.randn(1, 10)) prev_h = Variable(torch.randn(1, 20)) W_h = Variable(torch.randn(20, 20)) W_x = Variable(torch.randn(20, 10))
i2h = torch.mm(W_x, x.t()) h2h = torch.mm(W_h, prev_h.t()) next_h = i2h + h2h next_h = next_h.tanh()
next_h.backward(torch.ones(1, 20))
Add
MMMM
Tanh
PyTorch Autograd
![Page 24: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/24.jpg)
Neural Networks
![Page 25: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/25.jpg)
Neural Networks
![Page 26: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/26.jpg)
Neural Networks
![Page 27: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/27.jpg)
Optimization packageSGD, Adagrad, RMSProp, LBFGS, etc.
![Page 28: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/28.jpg)
Work items in practice
Writing Dataset loaders
Building models Implementing Training loop
Checkpointing models
Interfacing with environments
Dealing with GPUsBuilding optimizers Building
Baselines
![Page 29: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/29.jpg)
Work items in practice
Writing Dataset loaders
Building models Implementing Training loop
Checkpointing models
Interfacing with environments
Dealing with GPUsBuilding optimizers Building
Baselines
Python + PyTorch - an environment to do all of this
![Page 30: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/30.jpg)
Writing Data Loaders
• every dataset is slightly differently formatted
![Page 31: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/31.jpg)
Writing Data Loaders
• every dataset is slightly differently formatted • have to be preprocessed and normalized differently
![Page 32: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/32.jpg)
• every dataset is slightly differently formatted • have to be preprocessed and normalized differently • need a multithreaded Data loader to feed GPUs fast enough
Writing Data Loaders
![Page 33: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/33.jpg)
PyTorch solution: • share data loaders across the community!
Writing Data Loaders
![Page 34: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/34.jpg)
PyTorch solution: • share data loaders across the community!
Writing Data Loaders
![Page 35: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/35.jpg)
PyTorch solution: • use regular Python to write Datasets:
leverage existing Python code
Writing Data Loaders
![Page 36: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/36.jpg)
PyTorch solution: • use regular Python to write Datasets:
leverage existing Python code Example: ParlAI
Writing Data Loaders
![Page 37: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/37.jpg)
PyTorch solution: •Code in practice
Writing Data Loaders
![Page 38: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/38.jpg)
PyTorch solution: •Code in practice
Core Philisophy Upcoming Features
Pain PointsResearch Workflows
Writing Data Loaders
![Page 39: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/39.jpg)
PyTorch solution: •Code in practice
Core Philisophy Upcoming Features
Pain PointsResearch Workflows
Writing Data Loaders
![Page 40: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/40.jpg)
Interfacing with environments
Cars Video gamesInternet
![Page 41: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/41.jpg)
Interfacing with environments
Cars Video gamesPretty much every environment provides a Python API
![Page 42: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/42.jpg)
Interfacing with environments
Cars Video gamesNatively interact with the environment directly
![Page 43: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/43.jpg)
Debugging• PyTorch is a Python extension
![Page 44: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/44.jpg)
Debugging• PyTorch is a Python extension • Use your favorite Python debugger
![Page 45: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/45.jpg)
Debugging• PyTorch is a Python extension • Use your favorite Python debugger
![Page 46: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/46.jpg)
Debugging• PyTorch is a Python extension • Use your favorite Python debugger • Use the most popular debugger:
![Page 47: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/47.jpg)
Debugging• PyTorch is a Python extension • Use your favorite Python debugger • Use the most popular debugger:
print(foo)
![Page 48: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/48.jpg)
Identifying bottlenecks• PyTorch is a Python extension • Use your favorite Python profiler
![Page 49: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/49.jpg)
Identifying bottlenecks• PyTorch is a Python extension • Use your favorite Python profiler: Line_Profiler
![Page 50: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/50.jpg)
Compilation Time• PyTorch is written for the impatient
![Page 51: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/51.jpg)
Compilation Time• PyTorch is written for the impatient • Absolutely no compilation time when writing your scripts whatsoever
![Page 52: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/52.jpg)
Compilation Time• PyTorch is written for the impatient • Absolutely no compilation time when writing your scripts whatsoever
• All core kernels are pre-compiled
![Page 53: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/53.jpg)
Distributed PyTorch• MPI style distributed communication • Broadcast Tensors to other nodes • Reduce Tensors among nodes - for example: sum gradients among all nodes
![Page 54: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/54.jpg)
VisualizationTensorBoard-PyTorch Visdom
https://github.com/lanpa/tensorboard-pytorchhttps://github.com/facebookresearch/visdom
![Page 55: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/55.jpg)
Ecosystem• Use the entire Python ecosystem at your will
![Page 56: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/56.jpg)
Ecosystem• Use the entire Python ecosystem at your will • Including SciPy, Scikit-Learn, etc.
![Page 57: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/57.jpg)
Ecosystem• Use the entire Python ecosystem at your will • Including SciPy, Scikit-Learn, etc.
![Page 58: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/58.jpg)
Ecosystem• A shared model-zoo:
![Page 59: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/59.jpg)
Ecosystem• A shared model-zoo:
![Page 60: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/60.jpg)
Ecosystem•Probabilistic Programming
http://pyro.ai/
github.com/probtorch/probtorch
![Page 61: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/61.jpg)
Ecosystem•Gaussian Processes
https://github.com/cornellius-gp/gpytorch
![Page 62: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/62.jpg)
Ecosystem•QRNN - 2x to 17x faster than LSTM
https://github.com/salesforce/pytorch-qrnn
![Page 63: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/63.jpg)
Ecosystem•CycleGAN, pix2pix
https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
![Page 64: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/64.jpg)
Ecosystem•Machine Translation
https://github.com/OpenNMT/OpenNMT-py https://github.com/facebookresearch/fairseq-py
![Page 65: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/65.jpg)
Ecosystem•AllenNLP http://allennlp.org/
![Page 66: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/66.jpg)
Ecosystem•Pix2PixHD
https://github.com/NVIDIA/pix2pixHD
![Page 67: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/67.jpg)
Ecosystem•Sentiment Discovery
https://github.com/NVIDIA/sentiment-discovery
![Page 68: What is PyTorch?on-demand.gputechconf.com/gtc/2018/presentation/s... · next_h.backward(torch.ones(1, 20)) Add MM MM Tanh PyTorch Autograd. Neural Networks. Neural Networks. Neural](https://reader034.vdocument.in/reader034/viewer/2022042810/5f993fcc42d76910f8270c18/html5/thumbnails/68.jpg)
Ecosystem•FlowNet2: Optical Flow Estimation with Deep Networks
https://github.com/NVIDIA/flownet2-pytorch