gentlest introduction to tensorflow
TRANSCRIPT
![Page 1: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/1.jpg)
Gentlest Intro to TensorflowKhor, @neth_6, re:Culture
In collaboration with (Sam & Edmund):
![Page 2: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/2.jpg)
Goal● Perform linear regression in its most basic form through simplest example
○ Predict house price with just single feature, i.e., house size
![Page 3: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/3.jpg)
What We Will Learn Together● 4 steps of Machine Learning:
○ Create model○ Define cost function○ Collect data○ Train
● 2 parts of Tensorflow:○ Model linear regression & cost function as Tensorflow graph○ Train Tensorflow graph with dataset
![Page 4: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/4.jpg)
Machine Learning
![Page 5: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/5.jpg)
Machine Learning PurposePredict something given a set of features that influences prediction:
● Predict house price given its size, location, configuration (features)
![Page 6: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/6.jpg)
Machine Learning TrainingBefore predicting, we have to:
● Choose/Create a model● Train the model to learn prediction with data
![Page 7: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/7.jpg)
Problem
![Page 8: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/8.jpg)
Predict House Price Given House Sizey = house price
x = house size (feature)
Solve:
● Given a house size, what should be the house price?
![Page 9: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/9.jpg)
Model
![Page 10: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/10.jpg)
What is the House Price?House Price, $
House Size, sqm
![Page 11: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/11.jpg)
What is the House Price?House Price, $
House Size, sqm
Without data you cannot predict!
![Page 12: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/12.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 60
We can answer house price for 10, 20, 30, 50, 60 sqm
![Page 13: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/13.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 60
What is the price for 40 sqm?
40
![Page 14: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/14.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 60
What is the price for 40 sqm?
40
Linear regression best-fit prediction line
![Page 15: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/15.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 60
What is the price for 40 sqm?
40
Linear regression best-fit prediction line
Answer
![Page 16: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/16.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 60
What is the price for 40 sqm?
40
Exponential best-fit prediction line
Answer
![Page 17: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/17.jpg)
Cost Function: Best-fit Prediction
![Page 18: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/18.jpg)
Minimize difference between predicted & actual values
![Page 19: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/19.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 6040
Linear regression best-fit prediction line
![Page 20: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/20.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 6040
Linear regression best-fit prediction line
Difference betweenprediction & actual
![Page 21: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/21.jpg)
What is the House Price?House Price, $
House Size, sqm10 20 30 50 6040
Exponential best-fit prediction line
Difference betweenprediction & actual
![Page 22: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/22.jpg)
Cost function: How different is your model from reality
Where:● y_: Actual value● y: Predicted value● **2: Power of 2
sum( (y_ - y)**2)
![Page 23: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/23.jpg)
Linear Regression
![Page 24: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/24.jpg)
Linear Regression
y = W.x + by: Predicted house price
x: House size from dataset
Find a good W, b
![Page 25: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/25.jpg)
W: Gradient (Steepness)
y = W1.x + b y = W2.x + b y = W3.x + b
W1 < W2 < W3
![Page 26: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/26.jpg)
b: Intersect
y = W.x + b1 y = W.x + b2 y = W.x + b3
b1 < b2 < b3
![Page 27: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/27.jpg)
Train
![Page 28: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/28.jpg)
Train: Find W, bW_best, b_best = 0, 0
Loop J times:
Choose current_W, current_b
If cost(current_W, current_b) < cost(W_best, b_best):
W_best, b_best = current_W, current_b
![Page 29: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/29.jpg)
Train: Find W, bW_best, b_best = 0, 0
Loop J times:
Choose current_W, current_b
If cost(current_W, current_b) < cost(W_best, b_best):
W_best, b_best = current_W, current_b
What is the best way to choose current_W, current_b?
![Page 30: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/30.jpg)
Finding Minimum Cost Function Cost
W, b
You start here
Best W, b
![Page 31: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/31.jpg)
How to Reach Minimum?
![Page 32: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/32.jpg)
Choosing Training Method● Random
○ Could take forever
● Gradient descent○ From current viewpoint, move towards direction where steepness is greatest
![Page 33: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/33.jpg)
Ready?Recap:
● Model: Linear regression, y = W.x + b● Cost: Least squared, cost = sum( (y_ - y)**2 )● Train: Gradient descent
![Page 34: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/34.jpg)
Step 1: Model y = W.x + b
![Page 35: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/35.jpg)
Tensorflow Modeltf.placeholder
● Hold data to be feed into the model● x = tf.placeholder(tf.float32, [None, 1])
tf.Variable
● Hold variables to be trained● W = tf.Variable(tf.zeros([1,1]))● b = tf.Variable(tf.zeros([1]))
One feature, house sizeOne output, house price
![Page 36: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/36.jpg)
Tensorflow Modeltf.matmul(x,W)
● Multiply 2 matrices● product = tf.matmul(x,W)
‘+’
● y = product + b● Expands to:
○ y = tf.matmul(x,W) + b = W.x + b Done!
![Page 37: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/37.jpg)
Step 2: Cost Function sum( (y_ - y)** 2)
![Page 38: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/38.jpg)
Best-fit: Minimize Difference Prediction and ActualActual values:
● y_ = tf.placeholder(tf.float32, [None, 1])
Minimize difference between actual and prediction:
● cost = tf.reduce_sum(tf.pow((y_-y), 2))
One output, house price
![Page 39: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/39.jpg)
Step 3: Data
![Page 40: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/40.jpg)
Faking Datafor i in range(100): // Create fake data for actual data xs = np.array([[i]]) ys = np.array([[2*i]])
● xs = 0, 1, 2, …., 99● ys = 0, 2, 4, …., 198● ys is always twice of xs● y = W.x + b where W = 2, b = 0 => Best fit: y = 2x
![Page 41: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/41.jpg)
Step 4: Train
![Page 42: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/42.jpg)
Train using Gradient DescentTrain using Gradient Descent with steps in 0.000001
● train_step = tf.train.GradientDescentOptimizer(0.00001).minimize(cost)
![Page 43: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/43.jpg)
Train Model● sess = tf.Session()● init = tf.initialize_all_variables()● sess.run(init)● steps = 100
Tensorflow Non-interactive ModeNothing is actually executed until sess.run(...)
![Page 44: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/44.jpg)
Go!!!!!for i in range(steps): # Create fake data for y = W.x + b where W = 2, b = 0 xs = np.array([[i]]) ys = np.array([[2*i]])
# Train feed = { x: xs, y_: ys } sess.run(train_step, feed_dict=feed) print("After %d iteration:" % i) print("W: %f" % sess.run(W)) print("b: %f" % sess.run(b))
![Page 45: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/45.jpg)
Result
![Page 46: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/46.jpg)
OutputAfter 0 iteration:W: 0.000000b: 0.000000After 1 iteration:W: 0.000040b: 0.000040……After 98 iteration:W: 1.997181b: 0.051121After 99 iteration:W: 1.997632b: 0.051126
After 100 iterations:W ~ 2.0 and b ~ 0.0
![Page 47: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/47.jpg)
Debugging
![Page 48: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/48.jpg)
When Gradient Descent Go Awry?Re-run:
sess.run(tf.initialize_all_variables()) train_step = tf.train.GradientDescentOptimizer(0.01).minimize(cost)for i in range(steps): ….
![Page 49: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/49.jpg)
OutputAfter 0 iteration:W: 0.000000b: 0.000000After 1 iteration:W: 0.040000b: 0.040000After 2 iteration:W: 0.195200b: 0.117600......After 46 iteration:W: 323066974878921632721935905090174976.000000b: 7019517401500716765746276968955904.000000After 47 iteration:W: -infb: -296803829357474433113896004853170176.000000
![Page 50: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/50.jpg)
Finding Minimum Cost Function Cost
W, b
You start here
Best W, b
![Page 51: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/51.jpg)
Finding Minimum Cost Function Cost
W, b
You start here
Best W, b
![Page 52: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/52.jpg)
Finding Minimum Cost Function Cost
W, b
You start here
Best W, b
![Page 53: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/53.jpg)
Finding Minimum Cost Function Cost
W, b
You start here
Best W, b
![Page 54: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/54.jpg)
Errors: Missing Placeholdersfeed = { x: xs, y: ys }sess.run(train_step, feed_dict=feed)● y should be y_
W tensorflow/core/common_runtime/executor.cc:1076] 0x229e510 Compute status: Invalid argument: You must feed a value for placeholder tensor 'Placeholder_1' with dtype float
[[Node: Placeholder_1 = Placeholder[dtype=DT_FLOAT, shape=[], _device="/job:localhost/replica:0/task:0/cpu:0"]()]]
![Page 55: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/55.jpg)
Placeholder Numbers
cost = tf.reduce_sum(tf.pow((y_-y), 2))
y = product + b
product = tf.matmul(x,W)
x = tf.placeholder(tf.float32, [None, 1])
y_ = tf.placeholder(tf.float32, [None, 1])
Placeholder_1
Placeholder_0
![Page 56: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/56.jpg)
Assigning Names to Placeholders
cost = tf.reduce_sum(tf.pow((y_-y), 2))
y = product + b
product = tf.matmul(x,W)
x = tf.placeholder(tf.float32, [None, 1])
y_ = tf.placeholder(tf.float32, [None, 1], name=‘y-input’)
Placeholder_1
Placeholder_0
![Page 57: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/57.jpg)
Errors: Wrong Shapefor i in range(steps): xs = np.array([i]) ys = np.array([[2*i]])
● xs should be np.array([[i]])
Traceback (most recent call last): File "<stdin>", line 7, in <module> File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 364, in run
tuple(subfeed_t.get_shape().dims)))ValueError: Cannot feed value of shape (1,) for Tensor u'Placeholder:0', which has shape (Dimension(None), Dimension(1))
![Page 58: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/58.jpg)
Placeholder Numbers
cost = tf.reduce_sum(tf.pow((y_-y), 2))
y = product + b
product = tf.matmul(x,W)
x = tf.placeholder(tf.float32, [None, 1])
y_ = tf.placeholder(tf.float32, [None, 1])
Placeholder_1
Placeholder_0
![Page 59: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/59.jpg)
Print: Simplest Debuggerprint (sess.run(product), feed_dict={ x: np.array([1]) })
File "<stdin>", line 1print (sess.run(product), feed_dict={ x: np.array([1]) })
^SyntaxError: invalid syntax>>> print (sess.run(product, feed_dict={ x: np.array([1]) }))Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 364, in run
tuple(subfeed_t.get_shape().dims)))ValueError: Cannot feed value of shape (1,) for Tensor u'Placeholder:0', which has shape (Dimension(None), Dimension(1))
Same error so must be it!
![Page 60: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/60.jpg)
Progressing to Tensorflow Tutorial
![Page 61: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/61.jpg)
Difference
Parameters Gentlest Tensorflow Tutorial Tensorflow Tutorial
Purpose Find house price (Float32) Classify image
y Each sample is 1 x 1; Single-outcome (house price).Feature value is Float32.
Each sample is 10 x 1: One-hot vector for 10 possible values. Vector element is binary.
x Each sample is 1 x 1: Single-feature (house size) 1 x 1. Feature value is Float32.
Each sample is 1 x 784: 784 features (28 x 28 bits for each image).Feature value is binary
W 1 x 1: Single-outcome, single-feature
784 x 10: one for each 784 bits and 10 probability its one of 10 possible values
0001000000
One-hot vector for image 4
![Page 62: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/62.jpg)
Difference
Parameters Gentlest Tensorflow Tutorial Tensorflow Tutorial
b 1 x 1: Bias for single-outcome 10 x 1: Bias for 10 possible values of on-hot vector
cost Square of difference Entropy-based
gradient descent batch size
Single sample x, y 100 samples each batch
![Page 63: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/63.jpg)
Summary Machine Learning4 steps:
● Build model (linear regression, logistic regression, neural networks, etc.)● Define cost function to determine how well current model fits actual● Collect and prepare data● Train model to tweak model based on cost function using data
![Page 64: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/64.jpg)
Summary of Tensorflow● Modelling
○ tf.zeros○ tf.placeholder○ tf.Variable○ tf.matmul○ tf.reduce_sum○ tf.pow○ tf.train.GradientDescentOptimizer(step_size).minimize(cost)
● Train○ init = tf.initialize_all_variables()○ sess = tf.session()○ sess.run(init)○ sess.run(train, feed_dict={ x: …, y: … })
![Page 65: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/65.jpg)
Questions● Why not just use numpy.dot methods instead of tf.matmul?
○ tf.matmul is a Tensorflow Operation, and Tensorflow can easily assign each parallelizable operation to an available nodes to speed up operations
![Page 66: Gentlest Introduction to Tensorflow](https://reader034.vdocument.in/reader034/viewer/2022042610/58ab6bec1a28abb54e8b4b9d/html5/thumbnails/66.jpg)
ReferencesCode: https://github.com/nethsix/gentle_tensorflow/blob/master/code/linear_regression_one_feature.py