![Page 1: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/1.jpg)
Martin Andrews, ML GDE
TensorFlow Extended (TFX)(and a little bit of TensorFlow Lite)
@ BigDataX Singapore : 13-July-2019
![Page 2: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/2.jpg)
Martin Andrews
Google Developer Expert, Machine Learning
Red Dragon AI, Singapore
![Page 3: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/3.jpg)
Outline● Machine Learning for Production
○ Part of a Bigger Picture ● How the components are joined together● What all the components do :
○ Data ingestion …○ … ? ...○ … to serving (and TFlite)
● Wrap-up
![Page 4: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/4.jpg)
In addition to training a model ...
ML Code
![Page 5: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/5.jpg)
… a production solution requires so much more
Configuration
Data Collection
Data Verification
Feature Extraction
Process Management Tools
Analysis Tools
Machine Resource Management
Serving Infrastructure
Monitoring
ML Code
![Page 6: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/6.jpg)
Tensorflow Extended (TFX)
![Page 7: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/7.jpg)
Powers Alphabet’s most important bets and products
Tensorflow Extended (TFX)
![Page 8: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/8.jpg)
Libraries
Components
TFX Production Components
Data Validation
Feature Engineering
Train Model
Data Ingestion
Validate Model
Push If Good
Serve Model
![Page 9: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/9.jpg)
What is a Component?
![Page 10: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/10.jpg)
What makes a Component
Coordinates job execution
Performs the work
Updates ml.metadata
![Page 11: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/11.jpg)
What makes a Component?
Well defined config
![Page 12: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/12.jpg)
What makes a Component?
![Page 13: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/13.jpg)
What makes a Component?
![Page 14: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/14.jpg)
TFX: Metadata StoreWhat does it contain?
![Page 15: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/15.jpg)
Type definitions of Artifacts and their Properties
What is in Metadata Store?
![Page 16: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/16.jpg)
Type definitions of Artifacts and their Properties
Execution Records (Runs) of Components
What is in Metadata Store?
![Page 17: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/17.jpg)
Type definitions of Artifacts and their Properties
Execution Records (Runs) of Components
Data Provenance Across All Executions
What is in Metadata Store?
![Page 18: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/18.jpg)
Metadata-Powered Functionality
![Page 19: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/19.jpg)
Metadata-Powered Functionality
Find out which data a model was trained on
![Page 20: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/20.jpg)
Metadata-Powered Functionality
Compare previous model runs
![Page 21: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/21.jpg)
Metadata-Powered Functionality
Carry-over state from previous model runs
![Page 22: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/22.jpg)
Metadata-Powered Functionality
Re-use previously computed outputs
![Page 23: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/23.jpg)
TFX Orchestration
![Page 24: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/24.jpg)
Bring your own OrchestratorFlexible runtimes run components in the proper order using orchestration systems such as Airflow or Kubeflow
![Page 25: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/25.jpg)
Airflow Kubeflow Pipelines
Orchestrators and DAGs
![Page 26: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/26.jpg)
All the Components
![Page 27: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/27.jpg)
![Page 28: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/28.jpg)
Component: ExampleGen
examples = csv_input(os.path.join(data_root, 'simple'))
example_gen = CsvExampleGen(input_base=examples)
Configuration
ExampleGen
Raw Data
Inputs and Outputs
CSV TF Record
Split TF Record Data
Training
Eval
![Page 29: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/29.jpg)
Component: StatisticsGen
statistics_gen = StatisticsGen(input_data=example_gen.outputs.examples)
Visualization
StatisticsGen
Data
ExampleGen
Statistics
ConfigurationInputs and Outputs
![Page 30: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/30.jpg)
Analyzing Data with TensorFlow Data Validation
![Page 31: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/31.jpg)
Component: SchemaGen
SchemaGen
Statistics
StatisticsGen
Schema
infer_schema = SchemaGen(stats=statistics_gen.outputs.output)
Visualization
ConfigurationInputs and Outputs
![Page 32: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/32.jpg)
Component: ExampleValidator
ExampleValidator
Statistics Schema
StatisticsGen SchemaGen
Anomalies Report
validate_stats = ExampleValidator( stats=statistics_gen.outputs.output, schema=infer_schema.outputs.output)
Visualization
ConfigurationInputs and Outputs
![Page 33: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/33.jpg)
Component: Transform
transform = Transform( input_data=example_gen.outputs.examples, schema=infer_schema.outputs.output, module_file=taxi_module_file)
for key in _DENSE_FLOAT_FEATURE_KEYS: outputs[_transformed_name(key)] = transform.scale_to_z_score( _fill_in_missing(inputs[key]))# ...
outputs[_transformed_name(_LABEL_KEY)] = tf.where( tf.is_nan(taxi_fare), tf.cast(tf.zeros_like(taxi_fare), tf.int64), # Test if the tip was > 20% of the fare. tf.cast( tf.greater(tips, tf.multiply(taxi_fare, tf.constant(0.2))), tf.int64))
# ...
Transform
Data Schema
Transform Graph
Transformed Data
ExampleGen SchemaGen
Trainer
Code
Code
ConfigurationInputs and Outputs
![Page 34: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/34.jpg)
Using TensorFlow Transform for Feature Engineering
![Page 35: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/35.jpg)
Using TensorFlow Transform for Feature Engineering
Training Serving
![Page 36: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/36.jpg)
![Page 37: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/37.jpg)
Component: Trainer
Trainer
Data Schema
Transform SchemaGen
Evaluator
Code
Transform Graph
Model Validator Pusher
Model(s)
Highlight: SavedModel Format
TensorFlow Serving
TensorFlow Model Analysis
Train, Eval, and Inference Graphs
SignatureDefEvalSavedModel
SavedModel
Inputs and Outputs
![Page 38: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/38.jpg)
Component: Trainer
trainer = Trainer( module_file=taxi_module_file, transformed_examples=transform.outputs.transformed_examples, schema=infer_schema.outputs.output, transform_output=transform.outputs.transform_output, train_steps=10000, eval_steps=5000, warm_starting=True)
Just TensorFlow :)
Trainer
Data Schema
Transform SchemaGen
Evaluator
Code
Transform Graph
Model Validator Pusher
Model(s)
Inputs and Outputs
Code
Configuration
![Page 39: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/39.jpg)
![Page 40: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/40.jpg)
![Page 41: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/41.jpg)
![Page 42: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/42.jpg)
Component: Evaluator
Evaluator
Data Model
ExampleGen Trainer
Evaluation Metrics
model_analyzer = Evaluator( examples=examples_gen.outputs.output, eval_spec=taxi_eval_spec, model_exports=trainer.outputs.output)
Visualization
ConfigurationInputs and Outputs
![Page 43: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/43.jpg)
Component: ModelValidator
● Validate using current eval data● “Next-day eval”, validate using unseen data
Model Validator
Data
ExampleGen Trainer
Validation Outcome
Model (x2)
model_validator = ModelValidator( examples=examples_gen.outputs.output, model=trainer.outputs.output, eval_spec=taxi_mv_spec)
ConfigurationInputs and Outputs
Configuration Options
![Page 44: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/44.jpg)
Component: Pusher
Validation Outcome
Pusher
ModelValidator
PusherPusherDeployment
Options
pusher = Pusher( model_export=trainer.outputs.output, model_blessing=model_validator.outputs.blessing, serving_model_dir=serving_model_dir)
Block push on validation outcome
Push destinations supported today● Filesystem (TensorFlow Lite, TensorFlow JS)● TensorFlow Serving
ConfigurationInputs and Outputs
![Page 45: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/45.jpg)
Serve the Model !
![Page 46: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/46.jpg)
![Page 47: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/47.jpg)
TensorFlow ServingProduction-Ready
Used for years at Google, millions of QPS
Scale in minutes
Dynamic version refresh
![Page 48: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/48.jpg)
TensorFlow ServingHigh-Performance
Low-latency
Request Batching
Traffic Isolation
![Page 49: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/49.jpg)
$ apt-get install tensorflow-model-server
$ tensorflow_model_server
--port=8501
--model_name=chicago_taxi
--model_base_path='/path/to/savedmodel'
Deploy a REST API for your model in minutes ..
$ docker run -p 8501:8501 \
-v '/path/to/savedmodel':/models/chicago_taxi
-e MODEL_NAME=chicago_taxi -t tensorflow/serving
... or locally on
your host ...
... using Docker ...
![Page 50: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/50.jpg)
What is TensorFlow Lite?
![Page 51: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/51.jpg)
TensorFlow Lite is a framework for deploying ML on mobile devices and embedded systems
![Page 52: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/52.jpg)
Have now deployed TensorFlow Lite in production.
More than 2B devices globally.
Source: https://medium.com/tensorflow/recap-of-the-2019-tensorflow-dev-summit-1b5ede42da8d
![Page 53: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/53.jpg)
Classification
Prediction
Recognition
Text to Speech
Speech to Text
Object detection
Object location
OCR
Gesture recognition
Facial modelling
Segmentation
Clustering
Compression
Super resolution
Translation
Voice synthesis
Video generation
Text generation
Audio generation
AudioImageSpeechText Content
![Page 54: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/54.jpg)
Easy to get started
1
Jump start
Use our pretrained models or retrain
2
Custom model
Deploy your custom model
3
Performance
Benchmark, validate & accelerate your models.
4
Optimize
Try our Model Optimization Toolkit
![Page 55: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/55.jpg)
TensorFlow Lite powers ML Kit
● ML Kit is out-of-the-box proprietary models that you can run on device
![Page 56: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/56.jpg)
Converting your model Custom Model
TensorFlow(estimator or keras)
Saved Model TF LiteModel
TF LiteConverter
![Page 57: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/57.jpg)
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)tflite_model = converter.convert()open("converted_model.tflite", "wb").write(tflite_model)
TensorFlow 1.X
![Page 58: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/58.jpg)
● Limited ops
● Unsupported semantics (i.e. control-flow in RNNs)
Conversion is sometimes hard
Custom Model
![Page 59: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/59.jpg)
Goal: As fast as possible on all hardware (CPU, GPU, DSP, NPU)
Get your models running as fast as possible
Performance
![Page 60: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/60.jpg)
CPU 83 ms
CPU 1.8x 47 ms
GPU 5.5x15 ms
EdgeTPU 42x2 ms
Inference performance
CPU w/ Quantization
GPU OpenGL Float16
Quantized Fixed-point
CPU on MobileNet V1
Pixel 3 - Single Threaded CPU
MobileNet V1
![Page 61: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/61.jpg)
Achieved by reducing the precision of weights and activations in your graph.
Quantization: Huge speedup and ~4x smaller size
Optimize
![Page 62: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/62.jpg)
import tensorflow as tf
converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
converter.optimizations = [tf.lite.Optimize.OPTIMIZE_FOR_SIZE]
tflite_quant_model = converter.convert()
Optimize
![Page 63: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/63.jpg)
TensorFlow Lite for microcontrollers
MCU
More than 150B microcontrollers exist globally today
![Page 64: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/64.jpg)
What are they?
● No operating system
● Tens of KB of RAM & Flash
● Only CPU, memory & I/O peripherals
Small computer on a single circuitIO
RAM CPU ROM
MCU
![Page 65: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/65.jpg)
Input
MCU
Is there any sound?
Class 1
Class 2
Output Input
MCU
Is that human speech?
Class 1
Class 2
OutputDeeper
Network
ApplicationProcessor
![Page 66: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/66.jpg)
TensorFlow Saved Model
TensorFlow Lite Flat Buffer Format
TensorFlow Lite Interpreter
TensorFlow Lite Micro Interpreter
TensorFlow Lite for microcontrollers
TensorFlow provides you with a
single framework to deploy on
Microcontrollers as well as phones
![Page 67: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/67.jpg)
Example models
● Speech model (20KB)
● Image classifier (250KB) [Coming Soon]
Available now on tensorflow.org
![Page 68: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/68.jpg)
Wrap-up
![Page 69: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/69.jpg)
TensorFlow Extended (TFX)
Out-of-the-box components for your production model needs
Flexible orchestration and metadata
Extensible with custom components
Visit us at https://tensorflow.org/tfx and show us how you’ve used and extended TFX!
![Page 70: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/70.jpg)
https://www.tensorflow.org/tfx
![Page 71: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/71.jpg)
TensorFlow Lite
Serve models on mobile and embedded devices
Key features : Optimisation for speed and size
Makes TensorFlow ecosystem much more compelling
Visit us at https://tensorflow.org/tfx and show us how you’ve used and extended TFX!
![Page 72: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/72.jpg)
![Page 73: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/73.jpg)
Deep Learning MeetUp GroupThe Group :
● MeetUp.com / TensorFlow-and-Deep-Learning-Singapore● > 3,900 members
The Meetings :
● Next : Date TBA, hosted at Google○ Something for Beginners○ Something from the Bleeding Edge○ Lightning Talks
![Page 74: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/74.jpg)
Deep Learning Courses in SingaporeJumpstart Course : Two days in-person + One day online
● Hands-on with real model code● Build your own Project
Other Modules:
● Advanced Computer Vision; Advanced NLP; Self-supervised ...
Each 'module' includes :
● In-depth instruction, by practitioners● 70%-100% funding via IMDA for SG/PR
![Page 75: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/75.jpg)
Red Dragon AI : Intern HuntOpportunity to do Deep Learning “all day”
Key Features :
● Work on something cutting-edge (+ publish!)● Location : Singapore (SG/PR FTW) and/or Remote
Action points :
● Need to coordinate timing…● Contact Martin or Sam via LinkedIn
![Page 76: TensorFlow Extended (TFX) - REDCAT · 7/13/2019 · Martin Andrews, ML GDE TensorFlow Extended (TFX) (and a little bit of TensorFlow Lite) @ BigDataX Singapore : 13-July-2019](https://reader035.vdocument.in/reader035/viewer/2022062506/5f0af5b37e708231d42e2f3d/html5/thumbnails/76.jpg)
Questions?