epic nhl goal celebration hack with a hue light show and real-time machine
TRANSCRIPT
Epic NHL goal celebration hackwith a hue light show andreal time machine learning
François Maillet, Datacratic
PyCon Canada - Toronto November 7th, 2015
@mailletf blog.francoismaillet.com
About me
• Head of Machine Learning at Datacratic in Montréal
• Working on Machine Learning Database (MLDB.ai)
• Studied computer science and ML at Université de Montréal
• Research on music related problems: playlist generation, auto tagging, recommendation
blog.francoismaillet.com/epic-celebrationFor code snippets
and all technical details
Can we take it further?
Detect goals automatically?
Flash Philips Hues to the music?
Image source: http://www.chinadaily.com.cn/life/images/attachement/jpg/site1/20120807/001ec9790963118abddc19.jpg
The machine learning problem
• Binary classification task • Classes
• positive: the Canadiens scored a goal • negative: the Canadiens did not score a goal
• P(Habs goal | some audio)
Creating the dataset
• Habs website has 4 minutes highlights of previous games
• Extracted audio to wav files of 10 highlights
• Created CSV file containing time at which goals occurred
Audio feature extraction
• Library: librosa(github.com/bmcfee/librosa)
• Used mel power spectrogram
Feature vector
• Vectorize MPS
• 2 second clips, frequencies up to 8KHz with 128 Mel bands, at a sampling rate of 22.5KHz
• MPS have a shape of 128×87 -> feature vector of 11,136 elements when vectorized
Selecting training examples
• Positive examples: Canadians goals • Negative examples:
• Goals against • 50 random segments per clip
Training a classifier
• Library: scikit-learn
• Models: logistic regression and SVM
• Performance: 0.97AUC, 0.63 F1-Score
Evaluation metricsWHAT HAPPENED MODEL SAID RESULT
Habs scored! GOAL! True positive :D
Bad guys scored GOAL! REALLY BAD false positive
No goal GOAL! False positive
Habs scored! No goal False negative
What we used
• SVM: more true positives, but did worst job on false positives
• Used logistic regression
• To trigger a goal: 5 positives votes out of the last 20 and votes cast. Votes YES if the P(goal) > 90%.
• Cross-validation results: 23 Habs goals detected, 11 not detected, 2 opposing team goals falsely detected and no other false positives.
Looking at Habs’ 2014-15 season stats
• Scored an average of 2.61 goals per game; got scored 2.24 times
• Can loosely expect: • not detect 0.84 Habs goal per game
• go off for a goal by the opposing team once every 4 games.
Two parts to the light show
• Play Le goal song, by L’Oreille Cassée
• Flash lights with phue library
• github.com/studioimaginaire/phue
• Package it behind rest API using bottle
Hooking up the live audio stream
• Soundflower (rogueamoeba.com/freebies/soundflower) • virtual audio device • allows audio to be passed between apps
• pyaudio (people.csail.mit.edu/hubert/pyaudio) • play and record audio in Python
Real-time setupAudio from
browser
Soundflower
HDMI audio output
pyaudio
/goal endpoint
last 2 seconds of
audio
ML model P(goal | audio) 10 times / sec
last 20 predictions
Decision heuristic
Putting together the Youtube video
• Library: matplotlib
• Generated each frame and saved to PNGs
• Stitched with ImageMagick
• Converted to video with FFmpeg
Future work
• Which team’s goalie is on screen?
• P(Habs Goal on screen | image)
• Generalize to the other Francophone station in Montréal