introduction to pytorch - college of computing · introduction to pytorch. outline deep learning...
TRANSCRIPT
![Page 1: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/1.jpg)
Introduction to PyTorch
![Page 2: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/2.jpg)
Outline
● Deep Learning○ RNN
○ CNN
○ Attention
○ Transformer
● Pytorch○ Introduction
○ Basics
○ Examples
![Page 3: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/3.jpg)
Introduction to PyTorch
![Page 4: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/4.jpg)
What is PyTorch?
● Open source machine learning library ● Developed by Facebook's AI Research lab● It leverages the power of GPUs● Automatic computation of gradients● Makes it easier to test and develop new ideas.
![Page 5: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/5.jpg)
Other libraries?
![Page 6: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/6.jpg)
Why PyTorch?
● It is pythonic- concise, close to Python conventions● Strong GPU support● Autograd- automatic differentiation● Many algorithms and components are already
implemented● Similar to NumPy
![Page 7: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/7.jpg)
Why PyTorch?
![Page 8: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/8.jpg)
Getting Started with PyTorchInstallation
Via Anaconda/Miniconda: condainstall pytorch-c pytorch
Via pip: pip3 install torch
![Page 9: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/9.jpg)
PyTorch Basics
![Page 11: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/11.jpg)
Tensors
Tensors are similar to NumPy’s ndarrays, with the addition being that Tensors can also be used on a GPU to accelerate computing.
Common operations for creation and manipulation of these Tensors are similar to those for ndarrays in NumPy. (rand, ones, zeros, indexing, slicing, reshape, transpose, cross product, matrix product, element wise multiplication)
![Page 12: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/12.jpg)
TensorsAttributes of a tensor 't':
● t= torch.randn(1)
requires_grad- making a trainable parameter
● By default False● Turn on:
○ t.requires_grad_()or○ t = torch.randn(1, requires_grad=True)
● Accessing tensor value:○ t.data
● Accessingtensor gradient○ t.grad
grad_fn- history of operations for autograd
● t.grad_fn
![Page 13: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/13.jpg)
Loading Data, Devices and CUDANumpy arrays to PyTorch tensors
● torch.from_numpy(x_train)● Returns a cpu tensor!
PyTorchtensor to numpy
● t.numpy()
Using GPU acceleration
● t.to()● Sends to whatever device (cudaor cpu)
Fallback to cpu if gpu is unavailable:
● torch.cuda.is_available()
Check cpu/gpu tensor OR numpyarray ?
● type(t)or t.type()returns ○ numpy.ndarray○ torch.Tensor
■ CPU - torch.cpu.FloatTensor■ GPU - torch.cuda.FloatTensor
![Page 14: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/14.jpg)
Autograd● Automatic Differentiation Package
● Don’t need to worry about partial differentiation, chain rule etc.
○ backward()does that
● Gradients are accumulated for each step by default:○ Need to zero out gradients after each update○ tensor.grad_zero()
![Page 15: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/15.jpg)
Optimizer and LossOptimizer
● Adam, SGD etc.● An optimizer takes the parameters
we want to update, the learning rate we want to use along with other hyper-parameters and performs the updates
Loss
● Various predefined loss functions to choose from
● L1, MSE, Cross Entropy
![Page 16: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/16.jpg)
ModelIn PyTorch, a model is represented by a regular Python class that inherits from the Module class.
● Two components○ __init__(self): it defines the parts that make up the model- in our case, two
parameters, a and b○ forward(self, x) : it performs the actual computation, that is, it outputs a prediction,
given the inputx
![Page 17: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/17.jpg)
PyTorch Example(neural bag-of-words (ngrams) text classification)
bit.ly/pytorchexample
![Page 18: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/18.jpg)
Overview
Sentence
Softmax
Cross Entropy
EmbeddingLayer
Linear Layer
Prediction
EvaluationTraining
![Page 19: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/19.jpg)
Design Model
● Initilaize modules.● Use linear layer here. ● Can change it to RNN,
CNN, Transformer etc.
● Foward pass
● Randomly initilaize parameters
![Page 20: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/20.jpg)
Preprocess
● Build and preprocess dataset
● Build vocabulary
![Page 21: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/21.jpg)
Preprocess
● Create batch ( Used in SGD )● Choose pad or not ( Using [PAD] )
● One example of dataset:
![Page 22: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/22.jpg)
Training each epoch
Iterable batchesBefore each optimization, make previous gradients zeros
Backforward propagation to compute gradients and update parameters
Forward pass to compute loss
After each epoch, do learning rate decay ( optional )
![Page 23: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/23.jpg)
Test processDo not need back propagation or parameter update !
![Page 24: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/24.jpg)
The whole training process
Print information to monitor the training process
● Use CrossEntropyLoss() as the criterion. The input is the output of the model. First do logsoftmax, then compute cross-entropy loss.
● Use SGD as optimizer.● Use exponential decay
to decrease learning rate
![Page 25: Introduction to PyTorch - College of Computing · Introduction to PyTorch. Outline Deep Learning RNN CNN Attention Transformer Pytorch Introduction Basics Examples. Introduction to](https://reader034.vdocument.in/reader034/viewer/2022042622/5f98a5a5d1adde114c7b97d8/html5/thumbnails/25.jpg)
Evaluation with testdataset or random news