703 introducingcoreml 04 d - devstreaming-cdn.apple.com · void convolutionlayer(int kernelwidth,...

127
#WWDC17 © 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple. Gaurav Kapoor, Core ML Michael Siracusa, Core ML Lizi Ottens, Core ML Introducing Core ML System Frameworks

Upload: others

Post on 21-May-2020

36 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

#WWDC17

© 2017 Apple Inc. All rights reserved. Redistribution or public display not permitted without written permission from Apple.

Gaurav Kapoor, Core ML Michael Siracusa, Core ML Lizi Ottens, Core ML

•Introducing Core ML

System Frameworks

Page 2: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 3: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 4: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 5: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 6: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Personalization Face Detection Emotion Detection

Search Ranking Machine Translation Image Captioning

Real Time Image Recognition Text Prediction Entity Recognition

Speaker Identification Music Tagging Text Summarization

Sentiment Analysis Handwriting Recognition Style Transfer

Page 7: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Personalization Face Detection Emotion Detection

Search Ranking Machine Translation Image Captioning

Real Time Image Recognition Text Prediction Entity Recognition

Speaker Identification Music Tagging Text Summarization

Sentiment Analysis Handwriting Recognition Style Transfer

Page 8: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Why?

Page 9: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Task Show all images of roses

Page 10: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Task Show all images of roses

// Use color if color == "reddish"

Page 11: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Task Show all images of roses

// Use color if color == "reddish"

// Use shapeif shape == ???

Page 12: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Task Show all images of roses

Machine Learning

Page 13: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Training

Page 14: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Training

+ Labels

Offline

Page 15: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Training

Learning Algorithm

+ Labels

Offline

Page 16: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Training

Learning Algorithm Model

+ Labels

Offline

Page 17: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Inference

Model

Page 18: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Inference

Model

Page 19: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Inference

Label:

Confidence:

Rose 95%

Model

Page 20: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Inference

Challenging!

Label:

Confidence:

Rose 95%

Model

Page 21: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX, int strideY, int numRows, int numCols, float* input, float* output, float* weights, int widthPadding, int heightPadding, float alpha, float beta) { memset(output, 0, ((numRows - kernelWidth + 2*widthPadding)/2 + 1) * ((numRows - kernelWidth + 2*widthPadding)/2 + 1)*outputFeatureChannels * sizeof(float)); for (int depthInd = 0; depthInd < outputFeatureChannels; depthInd++) { // loop over input (color) channels for (int colorInd = 0; colorInd < inputFeatureChannels; colorInd++) { int numRowsOut = (numRows - kernelWidth + 2*widthPadding)/strideX + 1; int numColsOut = (numCols - kernelHeight + 2*heightPadding)/strideY + 1; // loop over the pixels of the image for (int i=0; i < numRowsOut; i++) { for(int j=0; j < numColsOut; j++) { // loop over this kernel for(int m=0; m < kernelWidth; m++) { int mm = kernelWidth - 1 - m; for(int n=0; n < kernelHeight; n++) { int nn = kernelHeight - 1 - n; int ii = i + (m - kernelWidth/2); int jj = j + (n - kernelHeight/2); if( ii >= 0 && ii < numRows && jj >= 0 && jj < numCols) float weight = weights[nn + mm * kernelHeight + depthInd * kernelHeight * kernelWidth]; float value = input[jj + ii*numCols + colorInd*numCols*inputFeatureChannels] * weight; output[j + i*numColsOut + depthInd*numColsOut*numColsOut] += value; } } } } } } // loop and apply nonlinearity for (int i = 0; i < ((numRows - kernelWidth + 2*widthPadding)/strideX + 1) * ((numRows - kernelWidth + 2*widthPadding)/strideY + 1) * outputFeatureChannels; i++) { output[i] = alpha*tanh(beta*output[i]); } }

Page 22: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Challenges

Correctness

Performance

Energy Efficiency

Page 23: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•ML Frameworks

Page 24: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

ML Frameworks

Your app

Page 25: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

ML Frameworks

NLPVision

Your app

NEWDomain Specific Frameworks

Page 26: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

ML Frameworks

NLP

Core ML

Vision

Your app

NEW

NEW

Domain Specific Frameworks

ML Framework

Page 27: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

ML Frameworks

MPS

NLP

Accelerate

Core ML

Vision

Your app

NEW

NEW

Domain Specific Frameworks

ML Framework

ML Performance Primitives

Page 28: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Vision Framework

Accelerate MPS

Core ML

Vision NLP

Your app

Page 29: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Vision Framework

Accelerate MPS

Core ML

Vision NLP

Your app

Object Tracking

Page 30: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Vision Framework

Accelerate MPS

Core ML

Vision NLP

Your app

Object Tracking Face Detection

Page 31: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Natural Language Processing

Accelerate MPS

Core ML

Vision NLP

Your app

Page 32: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Natural Language Processing

Accelerate MPS

Core ML

Vision NLP

Your app

Pablo y yo ya regresamos de nuestras vacaciones en

Finlandia.

Language Identification

Language: Spanish

Language Identification

Page 33: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Natural Language Processing

Accelerate MPS

Core ML

Vision NLP

Your app

Pablo y yo ya regresamos de nuestras vacaciones en

Finlandia.

Language Identification

Language: Spanish

Language Identification Named Entity Recognition

Place Identified: Finland

Pablo and I are back from Finland

Place Identification

Page 34: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Core ML

Accelerate MPS

Core ML

Vision NLP

Your app

Page 35: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Core ML

Accelerate MPS

Core ML

Vision NLP

Your app

Music Tagging

Page 36: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Core ML

Accelerate MPS

Core ML

Vision NLP

Your app

Music Tagging Image Captioning

Page 37: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Accelerate and MPS

High performance math

Inference for custom ML models

Accelerate MPS

Core ML

Vision NLP

Your app

Page 38: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Run on Device

Page 39: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Run on Device

User Privacy

Page 40: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Run on Device

User Privacy Data Cost

Page 41: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Run on Device

User Privacy Server CostData Cost

Page 42: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Run on Device

User Privacy Always Available

24/7

Server CostData Cost

Page 43: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 44: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 45: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

ML Frameworks

MPS

NLP

Accelerate

Core ML

Vision

Your app

NEW

NEW

Domain Specific Frameworks

ML Framework

ML Performance Primitives

Page 46: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Michael Siracusa, Core ML

•Core ML

Page 47: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview

Page 48: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview•Models

Page 49: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview•Models•Development Flow

Page 50: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview •Models •Development Flow

Page 51: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 52: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 53: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Focus on the experience you are trying to enable

Page 54: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Simple

Page 55: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

PerformantSimple

Page 56: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

PerformantSimple Compatible

Page 57: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Unified inference API

Xcode integrationSimple

Page 58: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Fine tuned inference engines

Built on Accelerate and MetalPerformant

Page 59: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Public model format

Support for popular training librariesCompatible

Page 60: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview •Models •Development Flow

Page 61: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Model

Function learned from data

Observed inputs

Predicts outputs

?Model

photo: flowerType:

Page 62: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Underlying Function

Scene Classification Style TransferTranslation

Sentiment Analysis Handwriting Recognition

Music Tagging Predicting Text

Page 63: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Underlying Function

Scene Classification Style TransferTranslation

Sentiment Analysis Handwriting Recognition

Music Tagging Predicting Text

😃That was totally awesome Leo!

I love you mom 사랑해 엄마

7

Do you know the way to San Jose

Beach

Rock

Page 64: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Model Types

Scene Classification Style TransferTranslation

Sentiment Analysis Handwriting Recognition

Music Tagging Predicting Text

Page 65: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Model Types

Scene Classification Style TransferTranslation

Sentiment Analysis Handwriting Recognition

Music Tagging Predicting Text

Convolutional Neural Networks

Feed Forward Neural Networks

Recurrent Neural Networks

Support Vector MachinesTree Ensembles Generalized Linear Models

Page 66: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Focus on Use Cases

Scene Classification Style TransferTranslation

Sentiment Analysis Handwriting Recognition

Music Tagging Predicting Text

Page 67: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Core ML Model

Single document

Public format

CoreML in Depth Hall 3 Thursday 9:00AM

Page 68: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Where do models come from?

Page 69: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Sample Models https://developer.apple.com/machine-learning

Core ML models

Ready to use

Task specific

Explore!

Page 70: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Thriving communities

Popular ML libraries

Many models

Tap Into ML Community

Page 71: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Convert to Core ML

Core ML Tools

Page 72: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Convert to Core ML

Core ML Tools

Page 73: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Convert to Core ML

Core ML Tools

Page 74: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Convert to Core ML

Core ML Tools

Open Source

Page 75: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Convert to Core ML

CoreML in Depth Hall 3 Thursday 9:00AM

Core ML Tools

Open Source

Page 76: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Overview •Models •Development Flow

Page 77: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Model as Code

Page 78: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode

Model as Code

Page 79: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode

Model as Code

Page 80: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode Your App

Model as Code

Page 81: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode Your App

Model as Code

Page 82: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Development Flow

Lizi Ottens, Core ML

Page 83: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 84: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 85: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Getting the Model

Page 86: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Getting the Model

Page 87: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Getting the Model

Page 88: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Getting the Model

Convert

Page 89: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,
Page 90: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

rose

Page 91: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Demo •Image based flower identifier

Page 92: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

•Demo Recap •Image based flower identifier

Page 93: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Xcode integration

Page 94: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Xcode integration

Page 95: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Xcode integration

Page 96: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Xcode integration

Page 97: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Simple usage

let flowerModel = FlowerClassifier() if let prediction = try? flowerModel.prediction(flowerImage: image) { return prediction.flowerType }

Page 98: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Simple usage

let flowerModel = FlowerClassifier() if let prediction = try? flowerModel.prediction(flowerImage: image) { return prediction.flowerType }

Type of model abstracted

Page 99: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Demo Recap Simple usage

let flowerModel = FlowerClassifier() if let prediction = try? flowerModel.prediction(flowerImage: image) { return prediction.flowerType }

Type of model abstracted

Input/output strongly typed

Page 100: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Generated Source

class FlowerClassifierInput { var flowerImage: CVPixelBuffer }

class FlowerClassifierOutput { let flowerType: String let flowerTypeProbs: [String: Double] }

class FlowerClassifier { convenience init() func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 101: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Generated Source

class FlowerClassifierInput { var flowerImage: CVPixelBuffer }

class FlowerClassifierOutput { let flowerType: String let flowerTypeProbs: [String: Double] }

class FlowerClassifier { convenience init() func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 102: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Generated Source

class FlowerClassifierInput { var flowerImage: CVPixelBuffer }

class FlowerClassifierOutput { let flowerType: String let flowerTypeProbs: [String: Double] }

class FlowerClassifier { convenience init() func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 103: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Generated Source

class FlowerClassifierInput { var flowerImage: CVPixelBuffer }

class FlowerClassifierOutput { let flowerType: String let flowerTypeProbs: [String: Double] }

class FlowerClassifier { convenience init() func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 104: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

More Advanced Underlying API

class FlowerClassifier { convenience init() func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 105: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

More Advanced Underlying API

class FlowerClassifier { convenience init()

func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 106: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

More Advanced Underlying API

Programmatic access to model for power users

class FlowerClassifier { convenience init()

let model: MLModel func prediction(flowerImage: CVPixelBuffer) throws -> FlowerClassifierOutput }

Page 107: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

class MLModel { var modelDescription: MLModelDescription func prediction(from input: MLFeatureProvider) throws -> MLFeatureProvider }

MLModel

Page 108: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

class MLModel { var modelDescription: MLModelDescription func prediction(from input: MLFeatureProvider) throws -> MLFeatureProvider }

MLModel

Access to model description

Page 109: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

class MLModel { var modelDescription: MLModelDescription func prediction(from input: MLFeatureProvider) throws -> MLFeatureProvider }

MLModel

Access to model description

Flexibility in how input is provided

Page 110: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode Your App

Page 111: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Xcode Your App

Page 112: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Behind the Scenes Model compilation

Page 113: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Behind the Scenes Model compilation

Page 114: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Behind the Scenes Model compilation

Quick initialization

Page 115: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Behind the Scenes Model compilation

Quick initialization

Optimized prediction

Page 116: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Model Goals

Page 117: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

NEWReduce size

Model Goals

Page 118: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

NEWReduce size

Improve accuracy

Model Goals

Page 119: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

NEWReduce size

Improve accuracy

Decrease prediction times

Model Goals

Page 120: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Summary

Page 121: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Summary

Machine learning frameworks

Page 122: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Summary

Machine learning frameworks

Core ML

Page 123: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Summary

Machine learning frameworks

Core ML

Development flow in action

Page 124: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

More Informationhttps://developer.apple.com/wwdc17/703

Page 125: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Related Sessions

Natural Language Processing and your Apps Hall 3 Wednesday 9:00AM

Vision Framework: Building on Core ML Hall 2 Wednesday 3:10PM

Core ML in depth Hall 3 Thursday 9:00AM

Accelerate and Sparse Solvers Executive Ballroom Thursday 10:00AM

Using Metal 2 for Compute Grand Ballroom A Thursday 4:10PM

Page 126: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,

Labs

Core ML and Natural Language Processing Lab Technology Lab D Thu 11:00AM-3:30PM

Core ML & Natural Language Processing Lab Technology Lab D Fri 1:50AM-4:00PM

Page 127: 703 IntroducingCoreML 04 D - devstreaming-cdn.apple.com · void convolutionLayer(int kernelWidth, int kernelHeight, int inputFeatureChannels, int outputFeatureChannels, int strideX,