from synthetic data to the holy grail of ai€¦ · sensor time series analysis failure prediction...
TRANSCRIPT
From synthetic data to the holy grail of AIGENERATIVE MODELS
2
ABOUT ME
- Solution Architect @ NVIDIA - Supporting delivery of AI / Deep Learning solutions
- 10 years experience working on HPC (high performance computing) systems, signal processing, and Machine Learning
- My past experience:
- Northrop Grumman – Systems Engineer
- Exxon Mobil – HPC Programmer
Aleksandr Volkov – [email protected]
3
WHY DO NEURAL NETWORKS WORK?
4
NEURAL NETWORKS ARE NOT NEWAnd are disapintingly simple as an algorithm
5
SUPERVISED LEARNING
y= f(x)
Approximating complex functions
6
SMALL NEURAL NETWORKSUnderperform
Data
Perfo
rman
ce
20101990 2000
7
WHY?
8
DATA
Data
Perfo
rman
ce
20101990 2000 2017 2020
9
COMPUTEHistorically we never had large datasets or compute
1980 1990 2000 2010 2020
102
103
104
105
106
107
Single-threaded perf
1.5X per year
1.1X per yearTransistors(thousands)
GPU-Computing perf1.5X per year 1000X
By 2025
10
PERSPECTIVE
11
CONTEXT1.759 petaFLOPs in November 2009
12
CONTEXT2 petaFLOPs - today
13
CONTEXT TODAY200 petaFLOPs in November 2018
Achitecture:
9,216 POWER9 22-core CPUs
27,648 Nvidia Tesla V100 GPUs
Approaching 3.3 exaopsusing mixed precision
14
2016 – Baidu Deep Speech 2Superhuman Voice Recognition
2015 – Microsoft ResNetSuperhuman Image Recognition
2017 – Google Neural Machine TranslationNear Human Language Translation
100 ExaFLOPS8700 Million Parameters
20 ExaFLOPS300 Million Parameters
7 ExaFLOPS60 Million Parameters
To Tackle Increasingly Complex ChallengesNEURAL NETWORK COMPLEXITY IS EXPLODING
15
100 EXAFLOPS=
2 YEARS ON A DUAL CPU SERVER
16
NEURAL NETWORKS ARE NOT NEWRequire abundance of data and compute
Data
Perfo
rman
ce
20101990 20172000 2020
17
EXPLODING DATASETS
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., ... & Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409.
Logarithmic relationship between the dataset size and accuracy• Translation
• Language Models
• Character Language Models
• Image Classification
• Attention Speech Models
18
EXPLODING DATASETS
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., ... & Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409.
Logarithmic relationship between the dataset size and accuracy
19
MAKING COMPLEX PROBLEMS EASY
20
TRANSFORMING IMPOSSIBLE INTO EXPENSIVE
21
22
SUPERVISED LEARNING
y= f(x)
Approximating complex functions
23
HUGE OPPORTUNITY
24
HUGE CHALLENGE
25
EXPLODING DATASETS
Hestness, J., Narang, S., Ardalani, N., Diamos, G., Jun, H., Kianinejad, H., ... & Zhou, Y. (2017). Deep Learning Scaling is Predictable, Empirically. arXiv preprint arXiv:1712.00409.
Logarithmic relationship between the dataset size and accuracy
26
A BETTER WAY
27
GENERATIVE MODELS
y= f(x)
Reformulating the problem
Training setGenerate new data with the same statistics
28
WE NO LONGER TRAIN OUR MODEL TO SOLVE A NARROW TASK
29
WE TRAIN IT TO LEARN A PHENOMENON (ITS PROBABILITY DISTRIBUTION)
30
TEACHING MACHINES HOW TO COMPOSE HUMAN LANGUAGE
31
UNDERSTANDING LANGUAGEGenerating text
Transformer LM – 100M Parameters
32
UNDERSTANDING LANGUAGEGenerating text
Transformer LM – 5B Parameters
33
“Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT-2 along with sampling code. We are not releasing the dataset, training code, or GPT-2 model weights.”
“Better Language Models and Their Implications”, OpenAI
34
TEACHING MACHINES HOW TO COMPOSE MUSIC
35
UNDERSTANDING BEAUTYComposing music
36
TEACHING MACHINES HOW TO GENERATE CODE
37
UNDERSTANDING CODEGenerating computer code
38
TEACHING MACHINES HOW TO GENERATE OTHER FORMS OF HUMAN LANGUAGE
39
UNDERSTANDING DESIGN2D drawings to 3D sketches
Yao, Jiaxian, et al. "Interactive Design and Stability Analysis of Decorative Joinery for Furniture." ACM Transactions on Graphics (TOG) 36.2 (2017): 20.
40
UNDERSTANDING DESIGNModels from specification
Autodesk research
41
TEACHING MACHINES HOW TO SPEAK
43
TEACHINGS MACHINES HOW TO SING
44
UNDERSTANDING BEAUTYPerforming music
45
UNDERSTANDING BEAUTYPerforming music
46
TEACHINGS MACHINES ABOUT HUMANS
47
UNDERSTANDING IMAGENeural Network is computing most likely value of pixels
Sønderby, Casper Kaae, et al. "Amortised map inference for image super-resolution." arXiv preprint arXiv:1610.04490 (2016).
48
UNDERSTANDING THE SHAPE OF A FACEFrom a single image
Saito, Shunsuke, et al. "Photorealistic Facial Texture Inference Using Deep Neural Networks." arXiv preprint arXiv:1612.00523 (2016).
49
UNDERSTANDING A CONCEPT OF A FACEGeneration
50
ANIMATING A FACEAnd more
Karras, Tero, et al. "Audio-driven facial animation by joint end-to-end learning of pose and emotion." ACM Transactions on Graphics (TOG) 36.4 (2017): 94.
51
ANIMATING A FACETeaching the network to animate facial expressions
52
TEACHING MACHINES ABOUT MOTION
53
CHARACTER ANIMATIONTeaching the network to animate movement
54
CHARACTER ANIMATIONTeaching the network to animate movement
55
TEACHINGS MACHINES ABOUT THE VISUAL WORLD
56
UNDERSTANDING IMAGETeaching the network the physics of the world
57
UNDERSTANDING IMAGE QUALITY
http://people.ee.ethz.ch/~ihnatova/
58
UNDERSTANDING THE VISUAL PROPERTIES OF THE WORLD
59
UNDERSTANDING THE VISUAL PROPERTIES OF THE WORLD
60
UNDERSTANDING IMAGE GENERAL PROPERTIES
How should the image look like from a different angle
61
TEACHING MACHINES ABOUT STYLE AND BEAUTY
62
SEMANTIC STYLE TRANSFERUnderstanding common semantic part of the image
63
GENERATING IMAGES FROM DESCRIPTION
Reed, Scott, et al. "Parallel Multiscale Autoregressive Density Estimation." arXiv preprint arXiv:1703.03664 (2017).
64
NOT JUST THEORETICAL RESEARCHNvidia GameWorks
65
TEACHINGS MACHINES ABOUT THE PHYSICAL WORLD
66
AI – THE NEW INSTRUMENT FOR SCIENCENeed for general purpose compute
67
NEURAL NETWORKS FOR SCIENCE
AI-POWERED WEATHER PREDICTION
PLASMA FUSION APPLICATION EARTHQUAKE SIMULATION
7.815.7
125
0
20
40
60
80
100
120
140
V100 TFLOPS
FP64+ MULTI-PRECISION
FP16 Solver3.5x times faster
FP16/FP321.15x ExaOPS
FP16-FP21-FP32-FP6425x times faster
68
AI – THE NEW INSTRUMENT FOR SCIENCEFluid dynamics
69
TEACHINGS MACHINES ABOUT ECONOMY
70
UNDERSTANDING ECONOMICSGenerating stock movement
71
• Usually postings or documents that exhibit an unusual or rare attribute values, such as:
• Seldom used user accounts,• Reverse postings, corrections
„Global“ Accounting Anomalies
• Usually postings or documents that exhibit an unusual or rare attribute combination, for example:
• Unusual posting activities• Deviating user behavior
„Local“ Accounting Anomalies#
Feat
ure
2 (e
.g. L
ine-
Item
s)
# Feature 1 (e.g. Amount)
# Fe
atur
e 2
(e.g
. Lin
e-Ite
ms)
# Feature 1 (e.g. Posting Amount)
Understanding BusinessDifferentiating mistakes from fraud
72
TEACHINGS MACHINES ABOUT MACHINES
73
PREDICTIVE MAINTENANCEFIELD INSPECTIONFACTORY INSPECTION
AI FOR INDUSTRIAL APPLICATIONS
Quality InspectionFault Detection & Classification
Inventory Inspection
Condition Based MaintenanceRemaining Useful Life
Sensor Time Series AnalysisFailure Prediction
Wide range of applications