Deep Learning for Predictive Maintenance

Pawel MorkiszGTC 2017

www.relia-sol.pl

Agenda

• Problem• Introduction and notion of deep neural networks

o Convolutional layerso Residual networks (ResNet)

• One dimensional convolutional networks in failure predictiono Approacho Results and the best architecture

www.relia-sol.pl

The problem

Inefficient operations• unexpected downtimes, repairs• lower productivity and safety

Delayed timeline• costly delays, • missing critical deadlines,• damaged customer relationships

PdM aMarket• $4.9B by 2021, at CAGR of 28.4%

Substantial cost and safety hazards caused by machinery failure

www.relia-sol.pl

Problem setting

• Sensor data collected in predefined time intervals• Well specified failure records

Data collected through thousands of

sensors

Pattern recognitionindicate oncoming failure, malfunction or anomalies

Clear insights related to operations, services,

logistics, design

www.relia-sol.pl

Data format

Timestamp Sensor 1 Sensor 2 Sensor 3 Sensor 4 Sensor 5 Sensor 6 …. Failure?

01.02.2011 00:03 999,7 5,300 5,547 0,087 3491,7 -0,942 ... 0

01.02.2011 00:04 744,6 6,053 20,665 0,178 1436,9 -0,820 ... 0

01.02.2011 00:05 4,7 9,111 3,116 0,226 6151,9 -0,410 ... 0

01.02.2011 00:06 840,9 4,413 7,863 0,059 7759,8 -0,065 ... 0

01.02.2011 00:07 756,7 0,606 22,314 0,131 4474,9 -0,429 ... 0

01.02.2011 00:08 750,9 6,303 4,633 0,092 3664,1 -0,318 ... 0

01.02.2011 00:09 639,8 3,826 5,382 0,206 3999,1 -0,271 ... 0

01.02.2011 00:10 274,2 9,073 16,963 0,066 2834,0 -0,514 ... 0

01.02.2011 00:11 551,6 4,383 16,822 0,183 1808,3 -0,334 ... 0

01.02.2011 00:12 983,7 3,497 22,169 0,087 9260,7 -0,632 ... 0

01.02.2011 00:13 742,7 3,012 23,503 0,042 7537,9 -0,481 ... 0

01.02.2011 00:14 24,7 1,394 2,590 0,085 163,9 -0,048 ... 0

01.02.2011 00:15 568,9 5,846 4,161 0,133 8403,1 -0,909 ... 1

01.02.2011 00:16 329,2 8,313 7,152 0,006 5390,7 -0,456 ... 1

01.02.2011 00:17 269,1 9,835 3,013 0,098 2576,4 -0,908 ... 1

www.relia-sol.pl

Problem setting

…. Failure?

... 0

... 0

... 0

... 0

... 0

... 0

... 0

... 1

... 1

... 1

... 1

... 1

... 1

... 1

... 1

𝑻

• Determine the time horizon 𝑻for failure prediction

• Observations during pre-failure period marked as the distinguished class

• Failure records itself can be removed or not, depending on how much they differ from the rest of the set

• Binary classification - evaluation of probabilitythat observation precedes failure

Predictive maintenance - interpretation

www.relia-sol.pl

1 2 3 4 5

Time

Se

nso

rs

Predictive maintenance - interpretation

www.relia-sol.pl

1 2 3 4 5

Time

Se

nso

rs

www.relia-sol.pl

Evaluation

• Data set is divided into three partso model learning, o validation of hiper-parameterso final evaluation

• Model search criterion in selected class is the quality on the second set

• The quality criterion between classes is quality on the third set

• Chronological divisiono prevents ‘prediction of the past

using future’

Timestamp Sensor 1 …. Failure?

01.02.2011 00:03 999,7 ... 0

01.02.2011 00:04 744,6 ... 0

01.02.2011 00:05 4,7 ... 0

01.02.2011 00:06 840,9 ... 0

01.02.2011 00:07 756,7 ... 0

01.02.2011 00:08 750,9 ... 0

01.02.2011 00:09 639,8 ... 0

01.02.2011 00:10 274,2 ... 0

01.02.2011 00:11 551,6 ... 0

01.02.2011 00:12 983,7 ... 0

01.02.2011 00:13 742,7 ... 0

01.02.2011 00:14 24,7 ... 0

01.02.2011 00:15 568,9 ... 1

01.02.2011 00:16 329,2 ... 1

01.02.2011 00:17 269,1 ... 1

Learning

Validation

Test

www.relia-sol.pl

Evaluation

• 𝐵 – number of not predicted failures,𝐶 – number of false alarms

• industrial problem significantly different costs of 𝑩, 𝑪

• Class cost coefficient 𝑥(included in model training)

Real

False True

ClassifiedFalse A 𝑩

True 𝑪 D𝐸𝑟𝑟 = 𝑩

𝑥

1 + 𝑥+ 𝑪

1

1 + 𝑥

www.relia-sol.pl

Independence of the observations

Actual Series

Seasonal Component Breakdown

Trend Component

Remainderrem

ain

de

r

tre

nd

se

aso

na

l

da

ta

time

• Sensor data o Collected cyclically,o Multidimensional time serieso Dependent!

• Many machine learning methodsrequire independence

• Data transformationso Decomposition (trend, periodicity,

etc.)o A lot of additional variables

www.relia-sol.pl

Convolution approach

32

323

1 1 1 0 0

0 1 1x1 1x0 0x1 4 3 4

0 0 1x0 1x1 1x0 2 4 3

0 0 1x1 1xo 0x1

0 1 1 0 0

Image ConvolvedFeature

www.relia-sol.pl

Convolutional Neural Network (CNN)

source:

• Weights sharing - less parameters• Better understanding of inherent data structure

www.relia-sol.pl

Residual networks

• Deeper architectures because the residual layers usually learn small, near zero values

• The winning architecture in many competitions

• Great stability improvementobserved

weight layer

weight layer

+

x

reluF(x)

H(x)=F(x)+x

relu

Identifyx

Sensor 1 Sensor 2 Sensor 3 ….

999,7 5,300 5,547 ...

744,6 6,053 20,665 ...

4,7 9,111 3,116 ...

840,9 4,413 7,863 ...

756,7 0,606 22,314 ...

750,9 6,303 4,633 ...

639,8 3,826 5,382 ...

274,2 9,073 16,963 ...

551,6 4,383 16,822 ...

983,7 3,497 22,169 ...

742,7 3,012 23,503 ...

24,7 1,394 2,590 ...

568,9 5,846 4,161 ...

329,2 8,313 7,152 ...

269,1 9,835 3,013 ...www.relia-sol.pl

Convolutional neural networks in failure prediction

• One dimensional filters • Applied only to columns,

i. e. on subsequent measurements from one sensor

www.relia-sol.pl

Network architecture

Input layer

4 x Conv (3 x 1 filter)

8 x Conv (3 x 1 filter)

16 x Conv (3 x 1 filter)

Dense layer (256 neurons)

Output layer (2 neurons)

www.relia-sol.pl

Techniques used

• Loss function taking into account large disproportions in the number of classes

• Batch normalization• L2 regularization• PReLU \ ReLU activation functions

www.relia-sol.pl

Results

Class\Real False True 𝐸𝑟𝑟

XGBoostFalse 50892 5

5.02True 29 49

DNN ResNet Average

False 47553 1.064.10

True 2893 52.94

DNN ResNet - Best

False 50391 00.06

True 55 54

𝑥 = 950𝐸𝑟𝑟 = 𝑩𝑥

1 + 𝑥+ 𝑪

1

1 + 𝑥

www.relia-sol.pl

Occupancy data set

• Attempt to use the same architecture

• Setting hyper-parameters on validation data

• Only changes:o Weights of classeso L2 regularization coefficient

www.relia-sol.pl

Occupancy data set

www.relia-sol.pl

Our solutions

• Cloud based IIoT solution• Fast and painless deployment and integration• Unlimited possibilities in:

o Adding new machines (with hierarchy)o Quick generatingpredictive modelso On-the-fly monitoring of assetso Identifying the causes of failures

• Immediate access to information worldwide

• Small plug & play predictive maintenancedevice

• Predictive model adjusted for the machine• Scalable• Onboard computations – no necessity of

constant Internet connection• Integrable with majority of industrial

transmission protocols• Low cost of purchase and deployment

The MindCloud, IIoT system

The EyeEdge device

www.relia-sol.pl

Our solutions

THANK YOU FOR YOUR ATTENTION!

Reliability Solutions Sp. z o.o.

Lublańska 34, 31-476 Kraków

Head-office: +48 (12) 394-11-21

Sales : +48 (12) 394-11-23

ACC: +48 (12) 627-77-15

R&D: +48 (12) 394-11-31

IT: +48 (12) 394-11-29

office@relia-sol.pl

We invite you to our booth 1132!