![Page 1: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/1.jpg)
February 11, 2018
Deep Learning Applications in Astronomy
clearnightsrthebest.com
Clécio R. Bom
![Page 2: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/2.jpg)
You will need:
-> Python (anaconda3)
-> Keras
-> Tensorflow
-> matplotlib
-> numpy
For the examples you will also need:
-> astropy
Google colabs (recommended) :
2
Lecture 01
Introdução ao aprendizado de máquina Algoritmo BackpropagationRedes Multi-Layer PerceptronConvolutional Neural Network.
Lecture 02
Training and convergenceQuality checksAlexNet,VGGInceptionResnet
Lecture 03
Auto Encoders. Generative Adversarial Networks (GAN).
Lecture 04
Region Based Convolutional Neural Networks (R-CNN).Long-Short Term Memory (LSTM)Reinforcement Learning
Program
![Page 3: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/3.jpg)
3
We will have several do-yourself examples in google collabs or in python notebooks. The examples and datasets required can be downloaded in
clearnightsrthebest.com
We have two tutors: - Luciana Olivia Dias, MSC. - Patrick Schubert.
They will be available from 13h-15h (1 p.m. - 3 p.m.).They can help with the examples.
General Announcements
![Page 4: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/4.jpg)
There is just too many data to analize (not enough woman/man power)
Automatize (lazyness? highter productivity)
Get intuitions, find patterns never seen before
If you like the idea of world ruled by robots
I want to make (tons of) money (outside academia) and live by the beach
Why go deep?
![Page 5: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/5.jpg)
main Strategies : Supervised or Unsupervised
Common applications: Classification, Regression
Semantic segmentation, data simulations, image enhancement, beat humans in games
Caveats: data hungry, not self-explanatory , “unhuman” errors, very specialized.
Would they ever be like humans? Shall I tell my lazy uncle to start a campaign against robots?
How Deep (and Shallow) Learning works
5
![Page 6: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/6.jpg)
6
RCNNs and Semantic SegmentationThe Tradicional CNNs are build to run on same size images only.This is an issue in many real life applications.So, we need region proposals.
Adapted from: arXiv:1504.08083v2Girshick et al 2015.
![Page 7: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/7.jpg)
Classification : CNNs for Strong Lensing Detection
The authors trained and validate the model on a set of 20,000 LSST-like mock observations including a range of lensed systems of various sizes and signal-to-noise ratios (S/N).
arXiv:1703.02642
![Page 8: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/8.jpg)
Regression: CNNs for Photo-z in SDSS
Competitive photo-zs using cut-outs arXiv:1806.06607CNN vs K-NN fitting
![Page 9: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/9.jpg)
9
Image Enhancement : Generative Adversarial Neural NetworksFirst proposed by Goodfellow et al. 2014 arXiv:1406.2661
GAN has been exploited to restore images, simulate images.Schawinski et al. 2017 used to deconvolve images beyond the deconvolution limit (arXiv:1702.00403v1).
![Page 10: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/10.jpg)
10
Data Simulations: Generative Adversarial Neural NetworksFirst proposed by Goodfellow et al. 2014 arXiv:1406.2661
GAN has been exploited to restore images, simulate images.
Mustafa et al. 2017 claims that can use GAN to simulate weak lensing convergence maps (arXiv:1706.02390v1).
![Page 11: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/11.jpg)
11
How sophisticated/hard is Deep Learning?Do I need lots of Math?
It is Intuitive?
It is a Lego?
It is alchemy?It is a ‘black box’?
Well, all depends on how far you want to go....
![Page 12: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/12.jpg)
12
Workflow - Supervised
Training• Architecture definition• Data augmentation• Batch size• Epochs
Evaluate the Model
Trained Model
Does it Make sense? Unclassified Real data
Classified data:Nice sims, or real data
Preprocessing
Train
Validation
Test*
Run trained model
Yes.No, need to go home rethink my life.
![Page 13: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/13.jpg)
13
Starting simple...
The most simple start ....
Developed by Frank Rosenblatt in the 1950s and 1960s
Binary output
![Page 14: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/14.jpg)
Starting simple...
![Page 15: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/15.jpg)
Building a NAND Gate
![Page 16: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/16.jpg)
Cool, but ...Easily gets lots of parameters, particularly if everything is connected to everything.
Small variations in the neurons can have strong impact in the output.
![Page 17: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/17.jpg)
Sigmoid Activation
![Page 18: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/18.jpg)
18
Gradient Descendent / How to optimize this!
Let the Cost function J(Ѳ) = Ѳ2
let’s start with Ѳ = 3After each iteration we use the update rule:
Problem : find Ѳ that minimizes the cost
![Page 19: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/19.jpg)
19
Gradient Descendent / How to optimize this!Problem : find Ѳ that minimizes the cost
Adapted from: mccormickml.com/2014/03/04/gradient-descent-derivation/
![Page 20: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/20.jpg)
20
Gradient Descendent / How to optimize this! Linear model: h(x) = Ѳ0 + Ѳ1*x.
![Page 21: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/21.jpg)
21
How Neural Nets can perform better than linear regression?
Neural networks can in principle model nonlinearities automatically, which you would need to explicitly model using transformations (for instance splines etc.) in linear regression.
The universal approximation theorem states that a feed-forward dense network with a single hidden layer containing a finite number of neurons can approximate continuous functions on compact subsets of Rn, with a few assumptions on the activation. Thus simple neural networks can represent a wide variety of functions when given appropriate parameters. However, it does not say a word about learnability of those parameters.
![Page 22: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/22.jpg)
Example adapted from:mattmazur.com/2015/03/17/a-step-by-step-backpropagation-example/
Let´s Backpropagate
![Page 23: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/23.jpg)
Let´s Backpropagate
![Page 24: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/24.jpg)
Moving forward ….
![Page 25: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/25.jpg)
Moving forward ….
![Page 26: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/26.jpg)
The total error
![Page 27: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/27.jpg)
Considering the chain rule ...
![Page 28: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/28.jpg)
Considering the chain rule ...
![Page 29: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/29.jpg)
Moving backwards !
Updating the weights
![Page 30: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/30.jpg)
30
Let’s play lego!
![Page 31: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/31.jpg)
Convolutional Neural NetworksWhat makes CNNs so special?
• Based on mammal visual cortex• Extract surrounding-depending high-order features.• Specially useful for
• Images • Time-dependent parameters (Speech recognition, Signal analysis)
![Page 32: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/32.jpg)
Types of Layers on CNNsConvLayers
• Based on Convolution• Linear Operators• Automatic (Visual) Feature Extractors.• Change size of input data.
Activation Layers• Add non-linearity• Commonly follow each ConvLayer.• Easy to implement & FAST to execute.
![Page 33: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/33.jpg)
Conv Layers
![Page 34: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/34.jpg)
Conv Layers
![Page 35: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/35.jpg)
Going Back to Convolution....
Stride and Padding
![Page 36: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/36.jpg)
Dropout and Pooling
Adapted from : https://ml-cheatsheet.readthedocs.io/en/latest/layers.html
![Page 37: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/37.jpg)
The simplest example I know
Adapted from :
from keras.datasets import mnistfrom keras.layers import Dense, Flatten, Conv2D, MaxPooling2Dfrom keras.models import Sequential
model = Sequential()model.add(Conv2D(32, kernel_size=(5, 5), strides=(1, 1), activation='relu', input_shape=input_shape))model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))model.add(Conv2D(64, (5, 5), activation='relu'))model.add(MaxPooling2D(pool_size=(2, 2)))model.add(Flatten())model.add(Dense(1000, activation='relu'))model.add(Dense(num_classes, activation='softmax'))
![Page 38: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/38.jpg)
batch_size = 128num_classes = 10epochs = 10
# input image dimensionsimg_x, img_y = 28, 28
# load the MNIST data set, which already splits into train and test sets for us(x_train, y_train), (x_test, y_test) = mnist.load_data()
model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test), callbacks=[history])score = model.evaluate(x_test, y_test, verbose=0)
The simplest example I know
![Page 39: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/39.jpg)
The Challenge:
Classify 200k images, between real data and simulations, in 48 hours for each of the two types (multiband or single band).
To test the algorithm we have 20k simulated images which contains all sorts of problems in the imaging system.
Each team developed different algorithms, mostly based in CNNs.
Classification Example: Strong Lensing Challenge
39
Metcalf et al. 2018arXiv:1802.03609
![Page 40: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/40.jpg)
40
Metcalf et al. 2018arXiv:1802.03609
The Bronze medal
![Page 41: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/41.jpg)
Example 1 - Lens Detect
![Page 42: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/42.jpg)
Example 1 - Lens Detect
![Page 43: Clécio R. BomLecture 01 Introdução ao aprendizado de máquina Algoritmo Backpropagation Redes Multi-Layer Perceptron Convolutional Neural Network. Lecture 02 Training and convergence](https://reader033.vdocument.in/reader033/viewer/2022041914/5e68a38df3d2512a6e3defa8/html5/thumbnails/43.jpg)
February 11, 2018
Deep Learning Applications in Astronomy
clearnightsrthebest.com
Clécio R. Bom