deep learning in python - amazon s3 · deep learning in python example as seen by linear regression...

DEEP LEARNING IN PYTHON

Introduction to deep learning

Deep Learning in Python

Imagine you work for a bank● You need to predict how many transactions each customer

will make next year

Deep Learning in Python

Example as seen by linear regressionAge

Bank Balance

Retirement Status

…

Number of Transactions

Deep Learning in Python

Example as seen by linear regression

Predicted Transactions

Bank Balance

Retired

Not Retired

Model with no interactions

Predicted Transactions

Bank Balance

Retired

Not Retired

Model with interactions

Deep Learning in Python

Interactions● Neural networks account for interactions really well

● Deep learning uses especially powerful neural networks

● Text

● Images

● Videos

● Audio

● Source code

Deep Learning in Python

Course structure● First two chapters focus on conceptual knowledge

● Debug and tune deep learning models on conventional prediction problems

● Lay the foundation for progressing towards modern applications

● This will pay off in the third and fourth chapters

Deep Learning in Python

Build deep learning models with kerasIn [1]: import numpy as np

In [2]: from keras.layers import Dense

In [3]: from keras.models import Sequential

In [4]: predictors = np.loadtxt('predictors_data.csv', delimiter=',')

In [5]: n_cols = predictors.shape[1]

In [6]: model = Sequential() In [7]: model.add(Dense(100, activation='relu', input_shape = (n_cols,)))

In [8]: model.add(Dense(100, activation='relu')

In [9]: model.add(Dense(1))

Deep Learning in Python

Deep learning models capture interactions

Age

Bank Balance

Retirement Status

…

Number of Transactions

Deep Learning in Python

Interactions in neural network

Age

Number of Transactions

Income

# Accounts

Input LayerHidden Layer

Output Layer

DEEP LEARNING IN PYTHON

Let’s practice!

DEEP LEARNING IN PYTHON

Forward propagation

Course Title

Bank transactions example● Make predictions based on:

● Number of children

● Number of existing accounts

Deep Learning in Python

Forward propagation

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

5

1

9 # Transactions

# Accounts

# Children

Deep Learning in Python

Forward propagation

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

5

1

9

# Children

# Accounts

# Transactions

Deep Learning in Python

Forward propagation

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

5

1

9

# Children

# Accounts

# Transactions

Deep Learning in Python

Forward propagation

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

5

1

9

# Children

# Accounts

# Transactions

Course Title

Forward propagation● Multiply - add process

● Dot product

● Forward propagation for one data point at a time

● Output is the prediction for that data point

Deep Learning in Python

In [1]: import numpy as np

In [2]: input_data = np.array([2, 3])

In [3]: weights = {'node_0': np.array([1, 1]), ...: 'node_1': np.array([-1, 1]), ...: 'output': np.array([2, -1])}

In [4]: node_0_value = (input_data * weights['node_0']).sum()

In [5]: node_1_value = (input_data * weights['node_1']).sum()

Forward propagation code

2

-1

Input Hidden Layer Output

3 1

51 2

-11

1

Deep Learning in Python

In [6]: hidden_layer_values = np.array([node_0_value, node_1_value])

In [7]: print(hidden_layer_values) [5, 1]

In [8]: output = (hidden_layer_values * weights['output']).sum()

In [9]: print(output) 9

Forward propagation code

2

-1

Input Hidden Layer Output

3 1

5

9

1 2

-11

1

DEEP LEARNING IN PYTHON

Let’s practice!

DEEP LEARNING IN PYTHON

Activation functions

Deep Learning in Python

Linear vs Nonlinear Functions

Linear Functions Nonlinear Functions

Deep Learning in Python

Activation functions● Applied to node inputs to produce node output

Deep Learning in Python

Improving our neural network

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

5

1

9

Deep Learning in Python

Activation functions

3

21

1

-1

1

2

-1

Input Hidden Layer

Output

tanh(2+3)

9

tanh(-2+3)

Deep Learning in Python

ReLU (Rectified Linear Activation)Rectifier

Deep Learning in Python

Activation functionsIn [1]: import numpy as np

In [2]: input_data = np.array([-1, 2])

In [3]: weights = {'node_0': np.array([3, 3]), ...: 'node_1': np.array([1, 5]), ...: 'output': np.array([2, -1])}

In [4]: node_0_input = (input_data * weights['node_0']).sum()

In [5]: node_0_output = np.tanh(node_0_input)

In [6]: node_1_input = (input_data * weights['node_1']).sum()

In [7]: node_1_output = np.tanh(node_1_input)

In [8]: hidden_layer_outputs = np.array([node_0_output, node_1_output])

In [9]: output = (hidden_layer_output * weights['output']).sum()

In [10]: print(output) 1.2382242525694254

DEEP LEARNING IN PYTHON

Let’s practice!

DEEP LEARNING IN PYTHON

Deeper networks

Deep Learning in Python

Multiple hidden layersAge

5

32

4

4

-5

-1

1

2

2

-3

7

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

4

-5

-1

1

2

2

-3

7

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

4

-5

-1

1

2

2

-3

7

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

4

-5

-1

1

2

2

-3

7

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

4

-5

-1

1

2

2

-3

7

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

Calculate with ReLU Activation Function

5

Deep Learning in Python

Multiple hidden layersAge

5

32

4

Calculate with ReLU Activation Function

5

26

Deep Learning in Python

Multiple hidden layersAge

5

3

Calculate with ReLU Activation Function

5

26

4

-5

Deep Learning in Python

Multiple hidden layersAge

5

3

Calculate with ReLU Activation Function

5

26

4

-5

Deep Learning in Python

Multiple hidden layersAge

5

3

Calculate with ReLU Activation Function

5

26

4

-5

Deep Learning in Python

Multiple hidden layersAge

5

3

Calculate with ReLU Activation Function

5

26

4

-50

2

4

-1

1

2

2

7

-3

Deep Learning in Python

Multiple hidden layersAge

5

3

Calculate with ReLU Activation Function

5

26

4

-50

2

4

-1

1

2

2

7

-30

52

364

Deep Learning in Python

Representation learning● Deep networks internally build representations of pa"erns in

the data

● Partially replace the need for feature engineering

● Subsequent layers build increasingly sophisticated representations of raw data

Deep Learning in Python

Representation learning

Deep Learning in Python

Deep learning● Modeler doesn’t need to specify the interactions

● When you train the model, the neural network gets weights that find the relevant pa"erns to make be"er predictions

DEEP LEARNING IN PYTHON

Let’s practice!