design and development videos - anna tikhonova, gpu software … · 2017-06-09 · house. horse...
TRANSCRIPT
![Page 1: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/1.jpg)
#WWDC17
© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.
Anna Tikhonova, GPU Software Engineer
•Using Metal 2 for Compute • Session 608
Graphics and Games
![Page 2: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/2.jpg)
Metal 2 Ecosystem
Metal API and language
GPU Tools
MetalKit
Metal Performance Shaders
Metal 2
![Page 3: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/3.jpg)
Metal 2 Ecosystem
Metal API and language
GPU Tools
MetalKit
Metal Performance Shaders
Metal 2
![Page 4: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/4.jpg)
Metal Performance Shaders (MPS)
GPU accelerated primitives • Image Processing • Linear Algebra • Machine Learning — Inference Optimized for iOS
What’s New in Metal, Part 2 WWDC 2016
What’s New in Metal, Part 2 WWDC 2015
![Page 5: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/5.jpg)
Metal Performance Shaders (MPS)
GPU accelerated primitives • Image Processing • Linear Algebra • Machine Learning — Inference Optimized for iOS
What’s New in Metal, Part 2 WWDC 2016
What’s New in Metal, Part 2 WWDC 2015
NEW
and macOS
![Page 6: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/6.jpg)
•Image Processing
![Page 7: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/7.jpg)
Image Processing Primitives available in iOS 10
Convolution
Gaussian Blur
Box, Tent
Sobel
Morphology
Lanczos Resampling
Histogram
Equalization and Specification
Median
Thresholding
Transpose
Image Integral
Color Conversion
Gaussian Pyramid
![Page 8: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/8.jpg)
Image Processing New primitives
Image Keypoints
Bilinear Rescale
Image Statistics
Element-wise Arithmetic Operations • With broadcasting
NEW
![Page 9: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/9.jpg)
•Linear Algebra
![Page 10: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/10.jpg)
Linear Algebra New primitives
Matrix-Matrix Multiplication
Matrix-Vector Multiplication
Triangular Matrix Factorization and Linear Solvers
NEW
![Page 11: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/11.jpg)
Data Representations
MPSVector • Interprets data in MTLBuffer as a 1-dimensional array
![Page 12: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/12.jpg)
Data Representations
MPSVector • Interprets data in MTLBuffer as a 1-dimensional array
MPSMatrix • Interprets data in MTLBuffer as a rectangular array • Row-major order
![Page 13: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/13.jpg)
Data Representations
MPSVector • Interprets data in MTLBuffer as a 1-dimensional array
MPSMatrix • Interprets data in MTLBuffer as a rectangular array • Row-major order
MPSTemporaryMatrix • Allocated from MTLHeap • Use for most of your intermediate matrices
![Page 14: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/14.jpg)
MPSVector and MPSMatrix Input types
Single Precision Floating-Point
Half Precision Floating-Point
16-bit Signed Integer
8-bit Signed Integer
![Page 15: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/15.jpg)
Create a vector of size N
// Create a Metal buffer of length N let buffer = device.makeBuffer(length: N * MemoryLayout<Float32>.size)
// Create a vector descriptor let descriptor = MPSVectorDescriptor(length: N, dataType: .float32)
// Create a vector with descriptor let vector = MPSVector(buffer: buffer, descriptor: descriptor)
MPSVector Code example
![Page 16: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/16.jpg)
Create a vector of size N
// Create a Metal buffer of length N let buffer = device.makeBuffer(length: N * MemoryLayout<Float32>.size)
// Create a vector descriptor let descriptor = MPSVectorDescriptor(length: N, dataType: .float32)
// Create a vector with descriptor let vector = MPSVector(buffer: buffer, descriptor: descriptor)
MPSVector Code example
![Page 17: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/17.jpg)
Create a vector of size N
// Create a Metal buffer of length N let buffer = device.makeBuffer(length: N * MemoryLayout<Float32>.size)
// Create a vector descriptor let descriptor = MPSVectorDescriptor(length: N, dataType: .float32)
// Create a vector with descriptor let vector = MPSVector(buffer: buffer, descriptor: descriptor)
MPSVector Code example
![Page 18: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/18.jpg)
Create a vector of size N
// Create a Metal buffer of length N let buffer = device.makeBuffer(length: N * MemoryLayout<Float32>.size)
// Create a vector descriptor let descriptor = MPSVectorDescriptor(length: N, dataType: .float32)
// Create a vector with descriptor let vector = MPSVector(buffer: buffer, descriptor: descriptor)
MPSVector Code example
![Page 19: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/19.jpg)
// Get the recommended bytes per row value to use for sizing a Metal buffer let bytesPerRow = MPSMatrixDescriptor.rowBytes(forColumns: N, dataType: .float32)
// Create a Metal buffer with the recommended bytes per row let buffer = device.makeBuffer(length: M * bytesPerRow)
// Create a matrix descriptor let descriptor = MPSMatrixDescriptor(rows: M, columns: N, rowBytes: bytesPerRow, dataType: .float32)
// Create a matrix with descriptor let matrix = MPSMatrix(buffer: buffer, descriptor: descriptor)
Create a matrix with M rows and N columns
MPSMatrix Code example
![Page 20: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/20.jpg)
// Get the recommended bytes per row value to use for sizing a Metal buffer let bytesPerRow = MPSMatrixDescriptor.rowBytes(forColumns: N, dataType: .float32)
// Create a Metal buffer with the recommended bytes per row let buffer = device.makeBuffer(length: M * bytesPerRow)
// Create a matrix descriptor let descriptor = MPSMatrixDescriptor(rows: M, columns: N, rowBytes: bytesPerRow, dataType: .float32)
// Create a matrix with descriptor let matrix = MPSMatrix(buffer: buffer, descriptor: descriptor)
Create a matrix with M rows and N columns
MPSMatrix Code example
![Page 21: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/21.jpg)
// Get the recommended bytes per row value to use for sizing a Metal buffer let bytesPerRow = MPSMatrixDescriptor.rowBytes(forColumns: N, dataType: .float32)
// Create a Metal buffer with the recommended bytes per row let buffer = device.makeBuffer(length: M * bytesPerRow)
// Create a matrix descriptor let descriptor = MPSMatrixDescriptor(rows: M, columns: N, rowBytes: bytesPerRow, dataType: .float32)
// Create a matrix with descriptor let matrix = MPSMatrix(buffer: buffer, descriptor: descriptor)
Create a matrix with M rows and N columns
MPSMatrix Code example
![Page 22: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/22.jpg)
// Get the recommended bytes per row value to use for sizing a Metal buffer let bytesPerRow = MPSMatrixDescriptor.rowBytes(forColumns: N, dataType: .float32)
// Create a Metal buffer with the recommended bytes per row let buffer = device.makeBuffer(length: M * bytesPerRow)
// Create a matrix descriptor let descriptor = MPSMatrixDescriptor(rows: M, columns: N, rowBytes: bytesPerRow, dataType: .float32)
// Create a matrix with descriptor let matrix = MPSMatrix(buffer: buffer, descriptor: descriptor)
Create a matrix with M rows and N columns
MPSMatrix Code example
![Page 23: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/23.jpg)
Primitives
Matrix-Matrix and Matrix-Vector Multiplication • API modeled after standard BLAS GEMM and GEMV interfaces
Triangular Matrix Factorization and Linear Solvers • API modeled after standard LAPACK decomposition and solve interfaces
![Page 24: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/24.jpg)
// Example: Matrix-Matrix Multiply: C = A B
// Create matrices A, B and C let A = MPSMatrix(buffer: ABuffer, descriptor: MPSMatrixDescriptor(rows: M, columns: K, rowBytes: ARowBytes, dataType: .float32)) let B = MPSMatrix(buffer: BBuffer, descriptor: MPSMatrixDescriptor(rows: K, columns: N, rowBytes: BRowBytes, dataType: .float32)) let C = MPSMatrix(buffer: CBuffer, descriptor: MPSMatrixDescriptor(rows: M, columns: N, rowBytes: CRowBytes, dataType: .float32))
![Page 25: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/25.jpg)
// Example: Matrix-Matrix Multiply: C = A B
// Perform Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Create a Matrix-Matrix Multiplication kernel let mmKernel = MPSMatrixMultiplication(device: device, resultRows: M, resultColumns: N, interiorColumns: K)
// Encode kernel to the command buffer mmKernel.encode(commandBuffer: commandBuffer, leftMatrix: A, rightMatrix: B, resultMatrix: C)
// Tell GPU to start doing the work commandBuffer.commit()
![Page 26: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/26.jpg)
// Example: Matrix-Matrix Multiply: C = A B
// Perform Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Create a Matrix-Matrix Multiplication kernel let mmKernel = MPSMatrixMultiplication(device: device, resultRows: M, resultColumns: N, interiorColumns: K)
// Encode kernel to the command buffer mmKernel.encode(commandBuffer: commandBuffer, leftMatrix: A, rightMatrix: B, resultMatrix: C)
// Tell GPU to start doing the work commandBuffer.commit()
![Page 27: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/27.jpg)
// Example: Matrix-Matrix Multiply: C = A B
// Perform Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Create a Matrix-Matrix Multiplication kernel let mmKernel = MPSMatrixMultiplication(device: device, resultRows: M, resultColumns: N, interiorColumns: K)
// Encode kernel to the command buffer mmKernel.encode(commandBuffer: commandBuffer, leftMatrix: A, rightMatrix: B, resultMatrix: C)
// Tell GPU to start doing the work commandBuffer.commit()
![Page 28: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/28.jpg)
// Example: Matrix-Matrix Multiply: C = A B
// Perform Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Create a Matrix-Matrix Multiplication kernel let mmKernel = MPSMatrixMultiplication(device: device, resultRows: M, resultColumns: N, interiorColumns: K)
// Encode kernel to the command buffer mmKernel.encode(commandBuffer: commandBuffer, leftMatrix: A, rightMatrix: B, resultMatrix: C)
// Tell GPU to start doing the work commandBuffer.commit()
![Page 29: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/29.jpg)
Sample Code
MPSMatrixMultiplication https://developer.apple.com/library/content/samplecode/MPSMatrixMultiplicationSample
Triangular Matrix Factorization and Linear Solvers Coming soon
![Page 30: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/30.jpg)
•Machine Learning
![Page 31: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/31.jpg)
MPS
NLP
Accelerate
Core ML
Vision
Applications
Domain Specific Frameworks
ML Framework
ML Performance Primitives
Machine Learning at Apple Architecture
![Page 32: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/32.jpg)
What Is Deep Learning?
![Page 33: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/33.jpg)
![Page 34: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/34.jpg)
![Page 35: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/35.jpg)
panda
![Page 36: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/36.jpg)
![Page 37: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/37.jpg)
giraffe
manhorse
ocean
plant
skateboard
girl
lights
sunset bicycle
dogdress
ramp
house
![Page 38: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/38.jpg)
horse
giraffe
dog
rabbit
cat
Training to Classify Images
Training and Inference
![Page 39: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/39.jpg)
Training
horse
giraffe
dog
rabbit
cat
Training to Classify Images
horse
giraffe
dog
dog
horse
dog
cat
cat
cat rabbit
rabbit
rabbit
![Page 40: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/40.jpg)
Training
Training to Classify Images
cat
rabbit
horse
giraffe
dog
![Page 41: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/41.jpg)
Training
Training to Classify Images
cat
rabbit
horse
giraffe
dogTrained
Parameters
![Page 42: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/42.jpg)
Inference
Training to Classify Images
cat
rabbit
horse
giraffe
dogTrained
Parameters
![Page 43: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/43.jpg)
Inference
Training to Classify Images
cat
rabbit
horse
giraffe
dog cat
Input Image
CNN
Inference
![Page 44: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/44.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN)
What’s New in Metal, Part 2 WWDC 2016
![Page 45: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/45.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN) •Convolutional Neural Networks — New Primitives •Neural Network Graph API •Recurrent Neural Networks (RNN)
![Page 46: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/46.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN) •Convolutional Neural Networks — New Primitives •Neural Network Graph API •Recurrent Neural Networks (RNN)
![Page 47: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/47.jpg)
What Are Convolutional Neural Networks?
![Page 48: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/48.jpg)
Convolutional Neural Networks
Biologically-inspired, resemble the visual cortex
![Page 49: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/49.jpg)
Convolutional Neural Networks
Biologically-inspired, resemble the visual cortex
Hierarchical representation • Organized into a hierarchy of layers • Higher-level features are derived from lower-level features
![Page 50: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/50.jpg)
Convolutional Neural Networks
Biologically-inspired, resemble the visual cortex
Hierarchical representation • Organized into a hierarchy of layers • Higher-level features are derived from lower-level features
Think of a “feature” as a filter that filters data for that feature
![Page 51: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/51.jpg)
Convolutional Neural Networks Primitives available in iOS 10
Convolution
Pooling • Average • Max
Normalization • Cross-Channel • Local Contrast • Spatial
Fully-Connected
Softmax
Neuron • Linear • ReLU • Sigmoid • TanH • Absolute
![Page 52: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/52.jpg)
Convolutional Neural Networks Primitives available in iOS 10
Convolution
Pooling • Average • Max
Normalization • Cross-Channel • Local Contrast • Spatial
Fully-Connected
Softmax
Neuron • Linear • ReLU • Sigmoid • TanH • Absolute
![Page 53: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/53.jpg)
Convolution
Core building block
Recognizes features in input
![Page 54: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/54.jpg)
1-channel output1-channel input
1 filter 3 x 3
![Page 55: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/55.jpg)
1-channel output1-channel input
1 filter 3 x 3
![Page 56: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/56.jpg)
1-channel output
1 filter 3 x 3
1-channel input
![Page 57: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/57.jpg)
1-channel output
1 filter 3 x 3
1-channel input
![Page 58: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/58.jpg)
1-channel output
1 filter 3 x 3
1-channel input
![Page 59: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/59.jpg)
1-channel output
1 filter 3 x 3
1-channel input
![Page 60: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/60.jpg)
16-channel output40 x 40
3-channel input 40 x 40
16 5x5filters
![Page 61: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/61.jpg)
16-channel output40 x 40
3-channel input 40 x 40
3*16 5x5filters
![Page 62: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/62.jpg)
16-channel output40 x 40
3-channel input 40 x 40
3*16 5x5filters
![Page 63: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/63.jpg)
16-channel output40 x 40
3-channel input 40 x 40
3*16 5x5filters
![Page 64: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/64.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN) •Convolutional Neural Networks — New Primitives •Neural Network Graph API •Recurrent Neural Networks (RNN)
![Page 65: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/65.jpg)
New Convolution weight types
Binary and XNOR Convolution
Sub-Pixel Convolution
Dilated Convolution
Convolution Transpose
L2Norm Pooling
Dilated Max Pooling
Log Softmax
Convolutional Neural Networks New primitives
NEW
Resampling • Lanczos, Bilinear
Upsampling
Arithmetic Operators • Addition, Subtraction,
Multiplication, Division
New Neuron layers • Hard Sigmoid, SoftPlus,
SoftSign, ELU
![Page 66: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/66.jpg)
New Convolution weight types
Binary and XNOR Convolution
Sub-Pixel Convolution
Dilated Convolution
Convolution Transpose
L2Norm Pooling
Dilated Max Pooling
Log Softmax
Convolutional Neural Networks New primitives
NEW
Resampling • Lanczos, Bilinear
Upsampling
Arithmetic Operators • Addition, Subtraction,
Multiplication, Division
New Neuron layers • Hard Sigmoid, SoftPlus,
SoftSign, ELU
![Page 67: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/67.jpg)
Convolution Filter weight types
Single Precision Floating-Point
To reduce memory footprint and improve performance • Half Precision Floating-Point • 8-bit Integer • Binary
NEW
![Page 68: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/68.jpg)
Convolution Primitives
Standard
Binary and XNOR
Dilated
Sub-Pixel
Transpose
NEW
![Page 69: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/69.jpg)
Same operation as regular Convolution
Improved performance
Less memory
Binary and XNOR Convolution
Regular Convolution
Input Weights
0.15 0.12 0.23 0.31 -0.510.16 -0.70 0.85 -0.67 0.790.73 0.92 -0.63 0.72 -0.110.86 -0.66 -0.12 0.19 0.23
0.65 -0.78 -0.36 0.72 0.13
0.68 -0.09 0.21 -0.39 0.23
![Page 70: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/70.jpg)
Binary Convolution • Full-sized input, binary weights
Binary and XNOR Convolution
Regular Convolution
Binary Convolution
Input Weights
0.15 0.12 0.23 0.31 -0.510.16 -0.70 0.85 -0.67 0.790.73 0.92 -0.63 0.72 -0.110.86 -0.66 -0.12 0.19 0.23
0.15 0.12 0.23 0.31 -0.510.16 -0.70 0.85 -0.67 0.790.73 0.92 -0.63 0.72 -0.110.86 -0.66 -0.12 0.19 0.23
+1 -1 -1 +1 +1
+1 -1 +1 -1 +1
0.65 -0.78 -0.36 0.72 0.13
0.68 -0.09 0.21 -0.39 0.23
![Page 71: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/71.jpg)
Binary Convolution • Full-sized input, binary weights
XNOR Convolution • Binary input, binary weights
Regular Convolution
Binary Convolution
XNOR Convolution
Input Weights
Binary and XNOR Convolution
0.15 0.12 0.23 0.31 -0.510.16 -0.70 0.85 -0.67 0.790.73 0.92 -0.63 0.72 -0.110.86 -0.66 -0.12 0.19 0.23
0.15 0.12 0.23 0.31 -0.510.16 -0.70 0.85 -0.67 0.790.73 0.92 -0.63 0.72 -0.110.86 -0.66 -0.12 0.19 0.23
+1 +1 +1 +1 -1
+1 -1 +1 -1 +1+1 +1 -1 +1 -1
+1 -1 -1 +1 +1
+1 -1 -1 +1 +1
+1 -1 +1 -1 +1
+1 -1 -1 +1 +1
+1 -1 +1 -1 +1
0.65 -0.78 -0.36 0.72 0.13
0.68 -0.09 0.21 -0.39 0.23
![Page 72: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/72.jpg)
Dilated Convolution Comparison to regular convolution
Input Output
![Page 73: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/73.jpg)
Dilated Convolution Comparison to regular convolution
Input Output
![Page 74: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/74.jpg)
Dilated Convolution Comparison to regular convolution
Input Output
3 x 3 kernel
![Page 75: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/75.jpg)
Dilated Convolution Comparison to regular convolution
Input Output
3 x 3 kernel
![Page 76: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/76.jpg)
Dilated Convolution How it works
Input Output
3 x 3 kernel dilationFactorX = 2 dilationFactorY = 2
![Page 77: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/77.jpg)
Dilated Convolution How it works
Input Output
3 x 3 kernel dilationFactorX = 2 dilationFactorY = 2
![Page 78: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/78.jpg)
Sub-Pixel Convolution and Convolution Transpose
Commonly used for upscaling
![Page 79: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/79.jpg)
Upscaling Using a box filter
Output 2W x 2H
Input W x H
Fixed operation with a constant filter
![Page 80: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/80.jpg)
Upscaling Using a box filter
Output 2W x 2H
Input W x H
Fixed operation with a constant filter
![Page 81: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/81.jpg)
Output 2W x 2H
Upscaling Using a box filter
Input W x H
Fixed operation with a constant filter
![Page 82: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/82.jpg)
Sub-Pixel Convolution How it works
One-channel input W x H
One-channel output 2W x 2H
4 filters for 2x upscaling
Trained Parameters
![Page 83: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/83.jpg)
Sub-Pixel Convolution How it works
4 filters for 2x upscaling
One-channel output 2W x 2H
One-channel input W x H
![Page 84: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/84.jpg)
Sub-Pixel Convolution How it works
4 filters for 2x upscaling
Reshuffle
One-channel output 2W x 2H
One-channel input W x H
![Page 85: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/85.jpg)
Input W x H
Convolution Transpose How it works
![Page 86: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/86.jpg)
Input W x H
Convolution Transpose How it works
![Page 87: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/87.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 88: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/88.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 89: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/89.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 90: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/90.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 91: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/91.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 92: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/92.jpg)
Convolution Transpose How it works
Intermediate Result 2W x 2H
Output W x H
![Page 93: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/93.jpg)
New Convolution Primitives Example: colorizing black and white images
![Page 94: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/94.jpg)
New Convolution Primitives Example: colorizing black and white images
*Colorful Image Colorization, Richard Zhang, Phillip Isola, Alexei A. Efros, ECCV 2016, http://richzhang.github.io/colorization/
Colorization network*
ConvolutionDilated ConvolutionBatch NormalizationConvolution TransposeSoftMax
Input Output
![Page 95: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/95.jpg)
New Convolution Primitives Example: colorizing black and white images
*Colorful Image Colorization, Richard Zhang, Phillip Isola, Alexei A. Efros, ECCV 2016, http://richzhang.github.io/colorization/
Dilated Convolution—integrate wider global context
ConvolutionDilated ConvolutionBatch NormalizationConvolution TransposeSoftMax
Colorization network*
![Page 96: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/96.jpg)
ConvolutionDilated ConvolutionBatch NormalizationConvolution TransposeSoftMax
Colorization network*
New Convolution Primitives Example: colorizing black and white images
*Colorful Image Colorization, Richard Zhang, Phillip Isola, Alexei A. Efros, ECCV 2016, http://richzhang.github.io/colorization/
Dilated Convolution—integrate wider global context
Convolution Transpose—upscale output
![Page 97: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/97.jpg)
•Demo •Image colorization
![Page 98: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/98.jpg)
Perc
enta
ge
Impr
ovem
ent
0
20
40
iPhone 6S iPhone 7 Plus iPad Pro 9.7” iPad Pro 10.5"
Performance Improvements in iOS 11
Higher is better
Inception-v3 network
*Rethinking the Inception Architecture for Computer Vision, Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna, CVPR 2015, https://arxiv.org/abs/1512.00567
![Page 99: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/99.jpg)
Perc
enta
ge
Impr
ovem
ent
0
20
40
iPhone 6S iPhone 7 Plus iPad Pro 9.7” iPad Pro 10.5"
22%
Performance Improvements in iOS 11
Higher is better
22%
29%
Inception-v3 network
*Rethinking the Inception Architecture for Computer Vision, Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens, Zbigniew Wojna, CVPR 2015, https://arxiv.org/abs/1512.00567
21%
![Page 100: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/100.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN) •Convolutional Neural Networks — New Primitives •Neural Network Graph API •Recurrent Neural Networks (RNN)
![Page 101: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/101.jpg)
Neural Network Graph API Overview
Describe neural network using graph API
NEW
![Page 102: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/102.jpg)
Neural Network Graph API Overview
Describe neural network using graph APIConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
NEW
![Page 103: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/103.jpg)
Describe neural network using graph API
Neural Network Graph API Overview
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 104: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/104.jpg)
Describe neural network using graph API
Filter nodes — Operations
Neural Network Graph API Overview
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 105: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/105.jpg)
Describe neural network using graph API
Filter nodes — Operations
Image nodes — Data
Neural Network Graph API Overview
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 106: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/106.jpg)
Neural Network Graph API Ease of use
Compact representation
![Page 107: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/107.jpg)
Neural Network Graph API Ease of use
Compact representation
Save and restore across platforms (NSSecureCoding)
![Page 108: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/108.jpg)
Neural Network Graph API Ease of use
Compact representation
Save and restore across platforms (NSSecureCoding)
Initialize once, reuse
![Page 109: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/109.jpg)
Neural Network Graph API Ease of use
Compact representation
Save and restore across platforms (NSSecureCoding)
Initialize once, reuse
Execute graph on GPU with single call
![Page 110: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/110.jpg)
Neural Network Graph API Ease of use
Compact representation
Save and restore across platforms (NSSecureCoding)
Initialize once, reuse
Execute graph on GPU with single call
No intermediate images to manage, just input/output
![Page 111: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/111.jpg)
Neural Network Graph API Ease of use
Compact representation
Save and restore across platforms (NSSecureCoding)
Initialize once, reuse
Execute graph on GPU with single call
No intermediate images to manage, just input/output
Auto-configuration of image sizes, padding, centering
![Page 112: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/112.jpg)
Neural Network Graph API Ease of use
https://developer.apple.com/library/content/samplecode/MetalImageRecognition
Compact representation
Save and restore across platforms (NSSecureCoding)
Initialize once, reuse
Execute graph on GPU with single call
No intermediate images to manage, just input/output
Auto-configuration of image sizes, padding, centering
MetalImageRecognition code sample* — 4x less code with NN Graph API
![Page 113: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/113.jpg)
Easy to parallelize between CPU and GPU
Neural Network Graph API Deliver best performance
![Page 114: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/114.jpg)
Easy to parallelize between CPU and GPU
Fuse graph nodes
Neural Network Graph API Deliver best performance
![Page 115: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/115.jpg)
Neural Network Graph API Deliver best performance
Easy to parallelize between CPU and GPU
Fuse graph nodes
Execute graph nodes concurrently
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 116: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/116.jpg)
Neural Network Graph API Deliver best performance
Easy to parallelize between CPU and GPU
Fuse graph nodes
Execute graph nodes concurrently
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 117: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/117.jpg)
Neural Network Graph API Deliver best performance
Easy to parallelize between CPU and GPU
Fuse graph nodes
Execute graph nodes concurrently
Optimize away Concatenation nodes
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 118: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/118.jpg)
Neural Network Graph API Deliver best performance
Easy to parallelize between CPU and GPU
Fuse graph nodes
Execute graph nodes concurrently
Optimize away Concatenation nodes
NEW
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMaxConcatentationImage
![Page 119: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/119.jpg)
Create a MPSNNConvolutionNode with data source provider
Filter Nodes Convolution node
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
![Page 120: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/120.jpg)
Create a MPSNNConvolutionNode with data source provider
Filter Nodes Convolution node
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
![Page 121: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/121.jpg)
Create a MPSNNConvolutionNode with data source provider
Filter Nodes Convolution node
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
![Page 122: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/122.jpg)
Just-in-time loading and purging of weights data
Minimize memory footprint
Feeding Parameters to Convolution Layer
class MyWeights: NSObject, MPSCNNConvolutionDataSource { // Initialize the data source object init(file: String) {…}
public func load() -> Bool {…} public func descriptor() -> MPSCNNConvolutionDescriptor {…} public func weights() -> UnsafeMutableRawPointer {…} public func purge() {…} }
![Page 123: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/123.jpg)
Just-in-time loading and purging of weights data
Minimize memory footprint
Feeding Parameters to Convolution Layer
class MyWeights: NSObject, MPSCNNConvolutionDataSource { // Initialize the data source object init(file: String) {…}
public func load() -> Bool {…} public func descriptor() -> MPSCNNConvolutionDescriptor {…} public func weights() -> UnsafeMutableRawPointer {…} public func purge() {…} }
![Page 124: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/124.jpg)
Just-in-time loading and purging of weights data
Minimize memory footprint
Feeding Parameters to Convolution Layer
class MyWeights: NSObject, MPSCNNConvolutionDataSource { // Initialize the data source object init(file: String) {…}
public func load() -> Bool {…} public func descriptor() -> MPSCNNConvolutionDescriptor {…} public func weights() -> UnsafeMutableRawPointer {…} public func purge() {…} }
![Page 125: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/125.jpg)
// Example: create a graph func makeGraph() -> MPSNNImageNode {
}
conv1
pool1
conv2
fc1
pool2
conv3
pool3
conv4
fc2
![Page 126: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/126.jpg)
// Example: create a graph func makeGraph() -> MPSNNImageNode {
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
}
conv1
pool1
conv2
fc1
pool2
conv3
pool3
conv4
fc2
![Page 127: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/127.jpg)
// Example: create a graph func makeGraph() -> MPSNNImageNode {
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
let pool1 = MPSCNNPoolingMaxNode(source: conv1.resultImage, filterSize: 2)
}
conv1
pool1
conv2
fc1
pool2
conv3
pool3
conv4
fc2
![Page 128: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/128.jpg)
// Example: create a graph func makeGraph() -> MPSNNImageNode {
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
let pool1 = MPSCNNPoolingMaxNode(source: conv1.resultImage, filterSize: 2)
let conv2 = MPSCNNConvolutionNode(source: pool1.resultImage, weights: MyWeights(file:“conv2.dat”))
let pool2 = MPSCNNPoolingMaxNode(source: conv2.resultImage, filterSize: 2)
let conv3 = MPSCNNConvolutionNode(source: pool2.resultImage, weights: MyWeights(file:“conv3.dat”))
let pool3 = MPSCNNPoolingMaxNode(source: conv3.resultImage, filterSize: 2)
let conv4 = MPSCNNConvolutionNode(source: pool3.resultImage, weights: MyWeights(file:“conv4.dat”))
let fc1 = MPSCNNFullyConnectedNode(source: conv4.resultImage, weights: MyWeights(file:“fc1.dat”))
let fc2 = MPSCNNFullyConnectedNode(source: fc1.resultImage, weights: MyWeights(file:“fc2.dat”))
return fc2.resultImage }
conv1
pool1
conv2
fc1
pool2
conv3
pool3
conv4
fc2
![Page 129: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/129.jpg)
// Example: create a graph func makeGraph() -> MPSNNImageNode {
let conv1 = MPSCNNConvolutionNode(source: MPSNNImageNode(handle: nil), weights: MyWeights(file:“conv1.dat”))
let pool1 = MPSCNNPoolingMaxNode(source: conv1.resultImage, filterSize: 2)
let conv2 = MPSCNNConvolutionNode(source: pool1.resultImage, weights: MyWeights(file:“conv2.dat”))
let pool2 = MPSCNNPoolingMaxNode(source: conv2.resultImage, filterSize: 2)
let conv3 = MPSCNNConvolutionNode(source: pool2.resultImage, weights: MyWeights(file:“conv3.dat”))
let pool3 = MPSCNNPoolingMaxNode(source: conv3.resultImage, filterSize: 2)
let conv4 = MPSCNNConvolutionNode(source: pool3.resultImage, weights: MyWeights(file:“conv4.dat”))
let fc1 = MPSCNNFullyConnectedNode(source: conv4.resultImage, weights: MyWeights(file:“fc1.dat”))
let fc2 = MPSCNNFullyConnectedNode(source: fc1.resultImage, weights: MyWeights(file:“fc2.dat”))
return fc2.resultImage }
conv1
pool1
conv2
fc1
pool2
conv3
pool3
conv4
fc2
![Page 130: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/130.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 131: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/131.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 132: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/132.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 133: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/133.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 134: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/134.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 135: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/135.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 136: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/136.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
![Page 137: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/137.jpg)
// Example: execute graph on the GPU // Metal setup let device = MTLCreateSystemDefaultDevice()! let commandQueue = device.makeCommandQueue() let commandBuffer = commandQueue.makeCommandBuffer()
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.encode(to: commandBuffer, sourceImages: [input]) // Tell GPU to start executing work and wait until GPU work is done commandBuffer.commit() commandBuffer.waitUntilCompleted()
CPU GPU
time
encode task1
execute task1
encode task2
execute task2
Bubble
Bubble
Bubble
encode task2
execute task2Bubble
Bubble
Bubbleencode task2
![Page 138: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/138.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
![Page 139: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/139.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
![Page 140: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/140.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
![Page 141: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/141.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
![Page 142: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/142.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
![Page 143: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/143.jpg)
// Example: execute graph on the GPU asynchronously // Metal setup let device = MTLCreateSystemDefaultDevice()!
// Initialize graph let graph = MPSNNGraph(device: device, resultImage: makeGraph())
// Create input image let input = MPSImage(texture: texture, …)
// Encode graph let output = graph?.executeAsync(sourceImages: [input]) { resultImage, error in // check for error and use resultImage inside closure }
// Don’t wait, encode new GPU task
CPU GPU
time
encode task1
encode task3
encode task2
encode task4
encode task5
encode task6
execute task1
execute task2
execute task3
execute task4
execute task5
![Page 144: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/144.jpg)
•Demo •Inception-v3 using Neural Network Graph API
![Page 145: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/145.jpg)
Agenda
•Recap on Convolutional Neural Networks (CNN) •Convolutional Neural Networks — New Primitives •Neural Network Graph API •Recurrent Neural Networks (RNN)
![Page 146: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/146.jpg)
What Are Recurrent Neural Networks?
![Page 147: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/147.jpg)
CNN One - to - one
One input Image
![Page 148: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/148.jpg)
dog grass …
CNN One - to - one
CNN
Inference
One output Set of probabilities
One input Image
![Page 149: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/149.jpg)
CNN
Inference
RNN Sequences: one - to - many
![Page 150: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/150.jpg)
A black and white dog
laying in the grass
CNN
Inference Inference
RNN
RNN Sequences: one - to - many
One input Set of probabilities
Sequence of outputs Words / image caption
![Page 151: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/151.jpg)
RNN Sequences: many - to - many
Inference
RNNA black and white dog
laying in the grass
Sequence of inputs Sentence in English
![Page 152: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/152.jpg)
RNN Sequences: many - to - many
Чёрно-белая собака лежит на траве
Inference
RNNA black and white dog
laying in the grass
Mustan ja valkoisen
värinen koira makaa
ruohikolla
Sequence of inputs Sentence in English
Sequence of outputs Translated sentence
![Page 153: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/153.jpg)
Recurrent Neural Networks New primitives
Single Gate
Long Short-Term Memory (LSTM)
Gated Recurrent Unit (GRU)
Minimally Gated Unit (MGU)
NEW
![Page 154: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/154.jpg)
Recurrent Unit enables previous output to affect the output of subsequent iterations
Single Gate RNN
Recurrent Unit
Output
Input
![Page 155: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/155.jpg)
Built from Single Gate RNNs
Has an internal Memory Cell
Gates control information flow inside the LSTM and what is stored in the Memory Cell
LSTM
Long Short-Term Memory (LSTM)
Output
Input
![Page 156: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/156.jpg)
Built from Single Gate RNNs
Has an internal Memory Cell
Gates control information flow inside the LSTM and what is stored in the Memory Cell
LSTM
Long Short-Term Memory (LSTM)
Output
Input
MemoryCell
![Page 157: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/157.jpg)
LSTM
Output
Input
MemoryCell
LSTM Architecture
![Page 158: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/158.jpg)
LSTM
MemoryCell
LSTM Architecture
![Page 159: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/159.jpg)
LSTM
LSTM Architecture
Old Memory
New Memory
![Page 160: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/160.jpg)
LSTM
LSTM Architecture
Old Memory
New Memory*
Forget Gate
Previous Output M
Input M
What to keep from old memory
M
*
Matrix-Matrix or Matrix-Vector Multiply
Point-wise operations+
![Page 161: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/161.jpg)
LSTM
LSTM Architecture
Old Memory
New Memory
Cell Gate
Previous Output M
Input M
*
*
Input Gate
Previous Output M
Input M
Forget Gate
Previous Output M
Input M
How new input affects new memory
What to keep from old memory
M
*
Matrix-Matrix or Matrix-Vector Multiply
Point-wise operations+
![Page 162: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/162.jpg)
LSTM
LSTM Architecture
Old Memory
New Memory
Cell Gate
Previous Output M
Input M
*
*
Input Gate
Previous Output M
Input M
Forget Gate
Previous Output M
Input M
How new input affects new memory
What to keep from old memory
M
*
Matrix-Matrix or Matrix-Vector Multiply
Point-wise operations+
![Page 163: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/163.jpg)
LSTM
LSTM Architecture
Old Memory
New Memory
Cell Gate
Previous Output M
Input M
*
*
Input Gate
Previous Output M
Input M
Forget Gate
Previous Output M
Input M
How new input affects new memory
What to keep from old memory
M
*
Matrix-Matrix or Matrix-Vector Multiply
Point-wise operations+
+
![Page 164: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/164.jpg)
LSTM
LSTM Architecture
Output
Old Memory
New Memory
*Output
Gate
Previous Output M
Input M
Cell Gate
Previous Output M
Input M
*
*
Input Gate
Previous Output M
Input M
Forget Gate
Previous Output M
Input M
How new input affects new memory
How previous output, current input, new memory affect new output
What to keep from old memory
M
*
Matrix-Matrix or Matrix-Vector Multiply
Point-wise operations+
![Page 165: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/165.jpg)
// Example: Creating a LSTM RNN
// Create a LSTM layer descriptor let descriptor = MPSLSTMDescriptor() descriptor.inputFeatureChannels = inputSize descriptor.outputFeatureChannels = outputSize
// Create and initialize gate weights with trained parameters, using a data source provider // for just-in-time loading and purging of weights descriptor.forgetGateInputWeights = MyWeights(file:“forgetGateWeights.dat”)) descriptor.cellGateInputWeights = MyWeights(file:“cellGateWeights.dat”)) // Initialize the rest of the gates…
// Metal setup let device = MTLCreateSystemDefaultDevice()! // Also get commandQueue and commandBuffer
// Create a LSTM layer let layer = MPSRNNMatrixInferenceLayer(device: device, rnnDescriptor: descriptor)
![Page 166: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/166.jpg)
// Example: Creating a LSTM RNN
// Create a LSTM layer descriptor let descriptor = MPSLSTMDescriptor() descriptor.inputFeatureChannels = inputSize descriptor.outputFeatureChannels = outputSize
// Create and initialize gate weights with trained parameters, using a data source provider // for just-in-time loading and purging of weights descriptor.forgetGateInputWeights = MyWeights(file:“forgetGateWeights.dat”)) descriptor.cellGateInputWeights = MyWeights(file:“cellGateWeights.dat”)) // Initialize the rest of the gates…
// Metal setup let device = MTLCreateSystemDefaultDevice()! // Also get commandQueue and commandBuffer
// Create a LSTM layer let layer = MPSRNNMatrixInferenceLayer(device: device, rnnDescriptor: descriptor)
![Page 167: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/167.jpg)
// Example: Creating a LSTM RNN
// Create a LSTM layer descriptor let descriptor = MPSLSTMDescriptor() descriptor.inputFeatureChannels = inputSize descriptor.outputFeatureChannels = outputSize
// Create and initialize gate weights with trained parameters, using a data source provider // for just-in-time loading and purging of weights descriptor.forgetGateInputWeights = MyWeights(file:“forgetGateWeights.dat”)) descriptor.cellGateInputWeights = MyWeights(file:“cellGateWeights.dat”)) // Initialize the rest of the gates…
// Metal setup let device = MTLCreateSystemDefaultDevice()! // Also get commandQueue and commandBuffer
// Create a LSTM layer let layer = MPSRNNMatrixInferenceLayer(device: device, rnnDescriptor: descriptor)
![Page 168: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/168.jpg)
// Example: Creating a LSTM RNN
// Create a LSTM layer descriptor let descriptor = MPSLSTMDescriptor() descriptor.inputFeatureChannels = inputSize descriptor.outputFeatureChannels = outputSize
// Create and initialize gate weights with trained parameters, using a data source provider // for just-in-time loading and purging of weights descriptor.forgetGateInputWeights = MyWeights(file:“forgetGateWeights.dat”)) descriptor.cellGateInputWeights = MyWeights(file:“cellGateWeights.dat”)) // Initialize the rest of the gates…
// Metal setup let device = MTLCreateSystemDefaultDevice()! // Also get commandQueue and commandBuffer
// Create a LSTM layer let layer = MPSRNNMatrixInferenceLayer(device: device, rnnDescriptor: descriptor)
![Page 169: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/169.jpg)
// Example: Running a LSTM RNN on the GPU
// Create input and output data var inputSequence: [MPSMatrix] = [] var outputSequence: [MPSMatrix] = [] for i in 0..< N { // Matrix size is (1, inputSize), inputSize is number of columns inputSequence.append(MPSMatrix(…)) // Matrix size is (1, outputSize), outputSize is number of columns outputSequence.append(MPSMatrix(…)) }
// Submit work to GPU layer.encodeSequence(commandBuffer: commandBuffer, sourceMatrices: inputSequence, destinationMatrices: outputSequence, recurrentInputState: nil, recurrentOutputStates: nil) // Tell GPU to start executing work commandBuffer.commit()
![Page 170: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/170.jpg)
// Example: Running a LSTM RNN on the GPU
// Create input and output data var inputSequence: [MPSMatrix] = [] var outputSequence: [MPSMatrix] = [] for i in 0..< N { // Matrix size is (1, inputSize), inputSize is number of columns inputSequence.append(MPSMatrix(…)) // Matrix size is (1, outputSize), outputSize is number of columns outputSequence.append(MPSMatrix(…)) }
// Submit work to GPU layer.encodeSequence(commandBuffer: commandBuffer, sourceMatrices: inputSequence, destinationMatrices: outputSequence, recurrentInputState: nil, recurrentOutputStates: nil) // Tell GPU to start executing work commandBuffer.commit()
![Page 171: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/171.jpg)
// Example: Running a LSTM RNN on the GPU
// Create input and output data var inputSequence: [MPSMatrix] = [] var outputSequence: [MPSMatrix] = [] for i in 0..< N { // Matrix size is (1, inputSize), inputSize is number of columns inputSequence.append(MPSMatrix(…)) // Matrix size is (1, outputSize), outputSize is number of columns outputSequence.append(MPSMatrix(…)) }
// Submit work to GPU layer.encodeSequence(commandBuffer: commandBuffer, sourceMatrices: inputSequence, destinationMatrices: outputSequence, recurrentInputState: nil, recurrentOutputStates: nil) // Tell GPU to start executing work commandBuffer.commit()
![Page 172: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/172.jpg)
Example: Image Captioning Training
Training to Caption Images
![Page 173: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/173.jpg)
caption
caption
caption
Example: Image Captioning Training
Trained Parameters
Training to Caption Images
![Page 174: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/174.jpg)
caption
caption
caption
Example: Image Captioning Training
Generate image
caption
Determine what is depicted in
the image
CNN RNNTrained
Parameters
![Page 175: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/175.jpg)
Trained Parameters
Example: Image Captioning Inference
![Page 176: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/176.jpg)
CNN
Example: Image Captioning Inference
Generate image
caption
Determine what is depicted in
the image
RNNTrained
ParametersTrained
Parameters
![Page 177: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/177.jpg)
CNN
Example: Image Captioning Inference
Generate image
caption
Determine what is depicted in
the image
RNN
Trained Parameters control CNN layers
Trained Parameters control RNN gates
![Page 178: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/178.jpg)
Example: Image Captioning Inference
Generate image
caption
Determine what is depicted in
the image
CNN RNN
![Page 179: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/179.jpg)
Example: Image Captioning Inference
a man riding a wave on top of a
surfboardGenerate image
caption
Determine what is depicted in
the image
CNN RNN
![Page 180: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/180.jpg)
Example: Image Captioning Inference
Generate image
caption
Determine what is depicted in
the image
Inception-v3
LSTMMemory
Cell
*Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge, Oriol Vinyals, Alexander Toshev, Samy Bengio, Dumitru Erhan, IEEE Transactions on Pattern Analysis and Machine Intelligence, https://arxiv.org/abs/1609.06647
Image Captioning Network*
a man riding a wave on top of a
surfboard
![Page 181: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/181.jpg)
Inception-v3
LSTM
MemoryCell
Example: Image Captioning LSTM initialization phase
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 182: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/182.jpg)
LSTM
Inception-v3
MemoryCell
Example: Image Captioning LSTM initialization phase
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 183: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/183.jpg)
LSTM
Inception-v3
MemoryCell
Example: Image Captioning LSTM initialization phase
Feature vector
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 184: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/184.jpg)
LSTM
Inception-v3
Example: Image Captioning LSTM initialization phase
MemoryCell
Feature vector
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 185: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/185.jpg)
Example: Image Captioning Caption generation phase
LSTMMemory
Cell
Input Sentence start token
Output
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 186: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/186.jpg)
3 best one-word captions
3 best one-word captions
Example: Image Captioning Caption generation phase
LSTMMemory
Cell
Input Sentence start token
Output
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 187: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/187.jpg)
3 best one-word captions
3 best one-word captions
Example: Image Captioning Caption generation phase
LSTMMemory
Cell
Input Sentence start token
Output
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 188: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/188.jpg)
3 best two-word captions
3 best one-word captions
Example: Image Captioning Caption generation phase
LSTMMemory
Cell
Input Sentence start token
Output
LSTMMemory
Cell
3 best one-word captions
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 189: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/189.jpg)
3 best two-word captions
3 best N-word captions
LSTMMemory
Cell
End
3 best one-word captions
Example: Image Captioning Caption generation phase
LSTMMemory
Cell
Input Sentence start token
Output
LSTMMemory
Cell. . .
3 best one-word captions
ConvolutionPooling (Avg.)Pooling (Max.)Fully-ConnectedSoftMax
![Page 190: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/190.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
Iteration 2
Caption Probability
![Page 191: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/191.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
Iteration 2
Caption Probability
![Page 192: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/192.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
![Page 193: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/193.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
a man 0.385814
a person 0.136590
a surfer 0.116651
![Page 194: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/194.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
a man 0.385814
a person 0.136590
a surfer 0.116651
the man 0.014275
the surfer 0.012315
the young 0.003500
![Page 195: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/195.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
a man 0.385814
a person 0.136590
a surfer 0.116651
the man 0.014275
the surfer 0.012315
the young 0.003500
Iteration 1
Caption Probability
man 0.021986
a 0.862899
the 0.039906
![Page 196: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/196.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 3
Caption Probability
a man riding 0.115423
a man on 0.060142
a man is 0.048678
a person riding 0.041114
a person on 0.031153
a person in 0.014218
a surfer is 0.027462
a surfer riding 0.016631
a surfer in 0.015737
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
a man 0.385814
a person 0.136590
a surfer 0.116651
the man 0.014275
the surfer 0.012315
the young 0.003500
![Page 197: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/197.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 3
Caption Probability
a man riding 0.115423
a man on 0.060142
a man is 0.048678
a person riding 0.041114
a person on 0.031153
a person in 0.014218
a surfer is 0.027462
a surfer riding 0.016631
a surfer in 0.015737
Iteration 2
Caption Probability
man on 0.005290
man in 0.004869
man surfing 0.003914
a man 0.385814
a person 0.136590
a surfer 0.116651
the man 0.014275
the surfer 0.012315
the young 0.003500
![Page 198: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/198.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 4
Caption Probability
a man riding a 0.100079
a man riding on 0.008264
a man riding the 0.002604
a man on a 0.055997
a man on his 0.001288
a man on the 0.001211
a man is surfing 0.021393
a man is riding 0.015222
a man is on 0.003434
Iteration 3
Caption Probability
a man riding 0.115423
a man on 0.060142
a man is 0.048678
a person riding 0.041114
a person on 0.031153
a person in 0.014218
a surfer is 0.027462
a surfer riding 0.016631
a surfer in 0.015737
![Page 199: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/199.jpg)
Caption Generation
Top three captions: 1. 2. 3.
Iteration 4
Caption Probability
a man riding a 0.100079
a man riding on 0.008264
a man riding the 0.002604
a man on a 0.055997
a man on his 0.001288
a man on the 0.001211
a man is surfing 0.021393
a man is riding 0.015222
a man is on 0.003434
Iteration 3
Caption Probability
a man riding 0.115423
a man on 0.060142
a man is 0.048678
a person riding 0.041114
a person on 0.031153
a person in 0.014218
a surfer is 0.027462
a surfer riding 0.016631
a surfer in 0.015737
![Page 200: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/200.jpg)
Caption Generation
Top three captions: 1. a man riding a wave on top of a surfboard 2. a man on a surfboard riding a wave 3. a man riding a wave on a surfboard
![Page 201: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/201.jpg)
Caption Generation
Top three captions: 1. a man riding a wave on top of a surfboard 2. a man on a surfboard riding a wave 3. a man riding a wave on a surfboard
![Page 202: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/202.jpg)
•Demo •Image captioning — CNN + LSTM
![Page 203: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/203.jpg)
Summary
GPU accelerated primitives • Expanded support for Image Processing and Convolutional Neural Networks • Added support for Linear Algebra and Recurrent Neural Networks
Optimized for iOS and macOS
New Neural Network Graph API
![Page 204: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/204.jpg)
Related Sessions
Introducing Metal 2 Executive Ballroom Tuesday 1:50PM
Introducing Core ML Hall 3 Tuesday 3:10PM
VR with Metal 2 Hall 3 Wednesday 10:00AM
Vision Framework: Building on Core ML Hall 2 Wednesday 3:10PM
Core ML in depth Hall 3 Thursday 09:00AM
Accelerate and Sparse Solvers Executive Ballroom Thursday 10:00AM
Metal 2 Optimization and Debugging Grand Ballroom B Thursday 3:10PM
![Page 205: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/205.jpg)
Labs
Metal 2 Lab Technology Lab Friday 09:00AM–12:00PMConfirm session DO NOT delete this note
![Page 206: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/206.jpg)
More Informationhttps://developer.apple.com/wwdc17/608
![Page 207: Design and Development Videos - Anna Tikhonova, GPU Software … · 2017-06-09 · house. horse giraffe dog rabbit cat Training to Classify Images Training and Inference . Training](https://reader033.vdocument.in/reader033/viewer/2022042914/5f4eb9f3c746083c70560e15/html5/thumbnails/207.jpg)