anomaly detection in deep learning

Anomaly Detection in Deep Learning Adam Gibson Skymind - Reactive Meetup 2016 @ Google Tokyo

Upload: adam-gibson

Post on 11-Jan-2017

2.497 views

Category:

Data & Analytics

0 download

Report

Download

Embed Size (px):

TRANSCRIPT

Anomaly Detection in Deep Learning

Adam Gibson Skymind - Reactive Meetup 2016 @ Google Tokyo

What’s an “Anomaly?”● Abnormal Patterns in Data

● Fraud Detection - “Bad credit card Transactions”

● ALSO Fraud detection - Detecting fake locations with call

detail records

● Network Intrusion - Abnormal Activity in a network

● Broken Computers in a data center

Page 3: Anomaly detection in deep learning

Brief Case Studies - eg: Why am I up here?● Telco: http://blogs.wsj.com/cio/2016/03/14/orange-tests-deep-

learning-software-to-identify-fraud/

● Network Infrastructure: https://insights.ubuntu.

com/2016/04/25/making-deep-learning-accessible-on-

openstack/

http://blogs.wsj.com/cio/2016/03/14/orange-tests-deep-learning-software-to-identify-fraud/

Page 4: Anomaly detection in deep learning

Network Infra - Save time and Money avoiding Broken workloads by auto migration before it happens

Page 5: Anomaly detection in deep learning

Why Deep Learning?● Learns well from lots of data

● Own feature representation: Robust to noise and allows for

learning cross domain patterns

● Already applied in ads: Google itself invests lots in this same

kind of pattern recognition (targeting/relevance)

Page 6: Anomaly detection in deep learning

Techniques● Unsupervised - Use autoencoder reconstruction error and use moving averages

use dropout with a set time window

● Supervised - RNNs Learn from a set of yes/nos in a time series. RNNs can learn

from a series of time steps and predict when an anomaly is about to occur.

● Use streaming/minibatches (all neural nets can learn like this)

Page 7: Anomaly detection in deep learning

Some definitions● Reconstruction Error: Autoencoders can learn from

unsupervised pretraining and learn how to reconstruct data.

Minimize KL Divergence (the delta between two probability

distributions

● RNN/Time Series: See http://deeplearning4j.org/usingrnns

Page 8: Anomaly detection in deep learning

Production● Kafka/Spark Streaming/Flink/Apex

● Neural net works as consumer of streaming updates

● Data? Mostly log ingestion, could be video

Page 9: Anomaly detection in deep learning

Questions?Email: [email protected]

Twitter: agibsonccc

Github: agibsonccc

mailto:[email protected]

Page 10: Anomaly detection in deep learning

Upcoming talksHadoop Summit: San Jose http://hadoopsummit.org/san-jose/ourspeakers/

Real-Time Anomaly detection...Real-Time Anomaly detection using Deep Learning to predict robots’ failures Dr Jean-Laurent PHILIPPE, Senior HPC Technical Specialist, Intel DCG With

Deep Nearest Neighbor Anomaly Detection

DeepLog: Anomaly Detection and Diagnosis from System Logs ...€¦ · DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du, Feifei Li, Guineng Zheng,

Anomaly Detection and Automatic Labeling with Deep Learning

A Deep Learning Approach to Anomaly Detection in Nuclear ...cortex-h2020.eu/wp-content/uploads/2018/09/2018... · A Deep Learning Approach to Anomaly Detection ... the core, along

GeoTrackNet—A Maritime Anomaly Detector using Probabilistic … · 2019. 12. 3. · Index Terms—AIS, maritime surveillance, deep learning, anomaly detection, variational recurrent

A Uniﬁed Deep Learning Anomaly Detection and Classiﬁcation

Anomaly detection for imbalanced datasets with deep generative models · Anomaly Detection for imbalanced datasets with Deep Generative Models Nazly Rocio Santos Buitrago 1, Loek

Anomaly Detection Survey

Real-Time Anomaly Detection using LSTM Auto-Encoders with Deep Learning4J on Apache Spark

Deep Semi-Supervised Anomaly Detection - TU Kaiserslautern · anomaly detection, we derive a loss motivated by the idea that the entropy for the latent distribution of normal data

Anomaly Detection with Robust Deep Autoencoders€¦ · Anomaly Detection with Robust Deep Autoencoders Chong Zhou Worcester Polytechnic Institute 100 Institute Road Worcester, MA

A Unifying Review of Deep and Shallow Anomaly Detection

Deep Learning for Unsupervised Anomaly Detection in ...tuora/aarontuor/materials/...Deep Learning for Unsupervised Insider Threat Detection in Structured Cybersecurity Data Streams

Deep Anomaly Detection via Morphological Transformations

ANOMALY DETECTION AND SECURITY DEEP LEARNING …

Anomaly detection

Attention-Driven Loss for Anomaly Detection in Video …pengxi.me/wp-content/uploads/2020/09/2020-TCSVT...imbalance problem in deep learning for video anomaly detection. 2) To better

Deep Anomaly Detection - AiFrenzAI Friends]Deep Anomaly... · Deep Anomaly Detection Kang, Min-Guk [email protected] Jan. 16, 2019 1/47

BGP Anomaly Detection

Anomaly Detection using Deep Auto-Encoders | Gianmario Spacagna

Clustering Driven Deep Autoencoder for Video Anomaly Detectionjsyuan/papers/2020/ECCV... · Clustering Driven Deep Autoencoder for Video Anomaly Detection Yunpeng Chang 1, Zhigang

DeepLog: Anomaly Detection and Diagnosis from …lifeifei/papers/deeplog.pdfDeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du, Feifei Li, Guineng

dLSTM: a new approach for anomaly detection using deep ......anomaly detection models suited to all kinds of anomalies. Toaddressthisproblem,anomalydetectiontypicallyuses models based

Anomaly Detection Tutorial

Science of Anomaly detection - v4 Updated for HTM for IT · The#Science#of#Anomaly#Detection!! ! Page 3! Anomaly!Detection!inStreamingData" Early anomaly detection in streaming data

DeepLog: Anomaly Detection and Diagnosis from System Logs ...lifeifei/papers/dl_ccs.pdf · DeepLog: Anomaly Detection and Diagnosis from System Logs through Deep Learning Min Du,

Deep learning for anomaly detectionin multivariate time ......Deep learning for anomaly detection in multivariate time series data Keywords Deep Learning, Machine Learning, Anomaly

Robust, Deep and Inductive Anomaly Detectionusers.cecs.anu.edu.au/~akmenon/papers/robust_ae/robust_ae.pdf · Robust, Deep and Inductive Anomaly Detection 5 3.1 Robust (convolutional)

Anomaly Detection Technology Using BigGraph€¦ · Anomaly Detection Technology Using BigGraph ... detection Several anomaly detection techniques have been developed and applied

Deep Learning based Anomaly Detection in · 2020. 7. 20. · Deep Learning based Anomaly Detection in Multivariate Time-Series Data 1. Background Anomaly detection, also known as

Network Traffic Anomaly Detection and Characterizationcs.northwestern.edu/~ajb200/anomaly detection paper 1.0.pdf · Network Traffic Anomaly Detection and Characterization Aaron Beach

A survey of deep learning-based network anomaly detection

Online Anomalous Trajectory Detection with Deep Generative … · B. Anomaly Detection with Deep Representation Learning Due to the powerful capacity of extracting complex infor-mation

Anomaly Detection and Mitigation. Outline DoS and DDoS Anomaly Detection and Mitigation Systems Cisco DDoS Anomaly Detection and Mitigation Solutions