data by the bay 2016 - black magic: how to apply machine learning to real-world problems
TRANSCRIPT
![Page 1: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/1.jpg)
Black MagicHow to apply ML to real-world problems
![Page 2: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/2.jpg)
It is great tool for some purposes
ML is (Magic) Hammer
![Page 3: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/3.jpg)
“If all you have is a hammer, everything looks like a nail
![Page 4: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/4.jpg)
![Page 5: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/5.jpg)
I am Evion KimLead Machine Learning Engineer @ Mattermark
Senior Software Engineer /Data Scientist @ Linkedin
M.S. , Computer Science @ Stanford University
B.S., Computer Science @ KAIST
Hello!
![Page 6: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/6.jpg)
Today’s Talk◇ Machine Learning - the concept◇ Mattermark?◇ Funding Extraction Problem @ Mattermark
& Some Magic Spells
![Page 7: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/7.jpg)
Not about
# Deep academic technical knowledge about ML algorithms
# Data Infrastructure
About
# How to transform real-world problem into ML-problem
# Tips and tricks on Machine Learning based problem solving
This talk is...
![Page 8: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/8.jpg)
Machine LearningThe powerful hammer
1
![Page 9: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/9.jpg)
![Page 10: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/10.jpg)
def traditional(x): return x*(x+1)
Traditional wayy = x * (x+1)
![Page 11: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/11.jpg)
2 = 63 = 124 = 205 = 306 = 42
ML Wayy = x * (x+1)
Model
![Page 12: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/12.jpg)
![Page 13: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/13.jpg)
Data
DEEP LEARNING?Trained Model
![Page 14: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/14.jpg)
It is not SKYNET… at least not “yet”.
![Page 15: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/15.jpg)
It is toolthat can be used for some problems
![Page 16: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/16.jpg)
What is(just quick advertisement)
2
![Page 17: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/17.jpg)
![Page 18: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/18.jpg)
![Page 19: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/19.jpg)
![Page 20: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/20.jpg)
![Page 21: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/21.jpg)
![Page 22: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/22.jpg)
Case Study: Funding Extraction-And Dark Magic spells we learned
3
![Page 23: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/23.jpg)
Small(er) company
# Much smaller training data points
# Very high precision requirement.
Big(ger) company
# Millions of Millions of training data points
# Precision requirement: not that high
ML @ Big(ger) vs. Small(er)
![Page 24: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/24.jpg)
What’s the bottleneck?
# Scalability or Accuracy?
# Precision or Recall?
# Engineering or Machine Learning?
Spell 1: Know your Enemy
![Page 25: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/25.jpg)
~$156 BillionTotal VC funding in year 2015
~8,532VC funding events in year 2015
![Page 26: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/26.jpg)
Problem to solve
![Page 27: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/27.jpg)
Divide big chunky problem into smaller ML-solvable problems.
Spell 2: Slice and Dice
![Page 28: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/28.jpg)
Smaller Problems
Classify Funding Articles
Classify Funding
Sentences
Extract Funding Entities
Confidence Scorer
![Page 29: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/29.jpg)
Classify Funding Article
TF-IDF + SVM Classifier
NO
YES
![Page 30: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/30.jpg)
Analyze and understand the problem space you are working on.
Spell 3: Understand your Domain
![Page 31: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/31.jpg)
Amount/Series/Investors
...has closed a $3.5m Series A funding round led by Inter Capital, ...
Investors
Intel Capital led the round with participation from other investors that included Horizons Ventures
Amount/ Series
...has raised $3.5 million in Series A Funding
Funding Sentences Patterns
![Page 32: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/32.jpg)
Classify Funding Sentences
Word2Vec
+ Semantic Role Labeling (SRL)
+ Gradient Boosting Classsifier
![Page 33: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/33.jpg)
Regex Parsing
+ Named Entity Recognition
Extract Funding Entities
![Page 34: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/34.jpg)
Spell 4: ProbabilisticTrain and use the probabilistic models helps a lot sometimes.
![Page 35: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/35.jpg)
“What’s the probability of these extracted information to be correct?”
Confidence Scoring
0~1 probability score
![Page 36: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/36.jpg)
Spell 5: Human + MachineLet some part of the job get the help from mighty human-being
![Page 37: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/37.jpg)
Human Administration
![Page 38: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/38.jpg)
~$156 BillionTotal VC funding in year 2015
~8,532VC funding events in year 2015
![Page 39: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/39.jpg)
Spell 5: Human + MachineSpell 4: ProbabilisticSpell 3: Understand your domain
Spell 2: Slice and DiceSpell 1: Know your enemyML is powerful Hammer
Summary
![Page 40: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/40.jpg)
We are hiring!
![Page 41: Data By The Bay 2016 - Black Magic: How to apply Machine Learning to real-world problems](https://reader033.vdocument.in/reader033/viewer/2022051710/58f2936f1a28ab66528b45b1/html5/thumbnails/41.jpg)