simple classification using binary datadeanna/simpleclass.pdf · 2017-10-09 · option 1 : build...
TRANSCRIPT
![Page 1: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/1.jpg)
Simple Classification using Binary Data
Deanna NeedellMathematicsUCLA
Thanks:• NSF CAREER DMS#1348721• NSF BIGDATA DMS#1740325• MSRI NSF DMS#1440140• Alfred P. Sloan Fdn
![Page 2: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/2.jpg)
Joint work with
Rayan Saab (UCSD) Tina Woolf (CGU)
![Page 3: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/3.jpg)
So much data…
![Page 4: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/4.jpg)
So much data…
![Page 5: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/5.jpg)
So much data…
![Page 6: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/6.jpg)
So much data…
![Page 7: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/7.jpg)
How can we handle all this data?
Option 1 : Build bigger computing systemsv We need the resourcesv Fundamental limitationsv Wasteful (resources, energy, cost, …)
![Page 8: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/8.jpg)
How can we handle all this data?
3 MB of internet data transfer = boiling one cup of water(https://www.katescomment.com/energy-of-downloads/)
![Page 9: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/9.jpg)
How can we handle all this data?
Option 2 : Design more efficient data analysis methods
![Page 10: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/10.jpg)
Compressed sensing
v Need to solve highly underdetermined linear system
vA has a null-space!
![Page 11: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/11.jpg)
1-bit compressed sensingv Store only the first bit – the SIGN of each measurement
![Page 12: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/12.jpg)
1-bit compressed sensingv Store only the first bit – the SIGN of each measurement
![Page 13: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/13.jpg)
1-bit compressed sensingv Store only the first bit – the SIGN of each measurement
v Remedy: Use “dithers” to estimate the norm of x
![Page 14: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/14.jpg)
Moral:
vOne-bit (binary) data is as efficient as it gets
vIt may still contain enough “information” about the signal to perform inference tasks
![Page 15: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/15.jpg)
Problem: classification
![Page 16: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/16.jpg)
Problem: reality
(Tang etal. 2016)
![Page 17: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/17.jpg)
Background: SVM
![Page 18: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/18.jpg)
Background: SVM
![Page 19: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/19.jpg)
Background: Deep Learning
![Page 20: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/20.jpg)
Our goals
Design classification scheme that:
v Uses binary data
v Is simple and efficient
v Uses layers in an interpretable way
![Page 21: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/21.jpg)
Notation
v X : nxp training data matrix (data in columns)
v A : mxn (random) matrix
v Q = sign(AX) : mxp binary training data
v G : # of classes
v b : p training labels (1-G)
v L : # of layers in our design
![Page 22: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/22.jpg)
Main idea
v Each row of A corresponds to a hyperplane
v Each binary measurement Qij indicates on which side of the ith hyperplane data point Xj lies
v If we gather enough of this info for all the training data, we can use it to predict the class label for a new test point x
![Page 23: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/23.jpg)
Single layer
v All the hyperplane information in Q is enough
For a new point x, simply compare its sign pattern with those of the training points and choose the label it matches the most often
x
![Page 24: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/24.jpg)
Multiple layers
v What about?
For ANY hyperplane, there are both red and blue on at least one side of it
x
![Page 25: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/25.jpg)
Multiple layers
v Now PAIRS of hyperplanes do the trick
x
For a new point x, simply compare its sign pattern for hyperplane PAIRS with those of the training points and choose the label it matches the most often
![Page 26: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/26.jpg)
Multiple layers
What about higher dimensions?
v We continue this strategy to build layers, the lthlayer corresponding to l–tuples of hyperplanes
v For simplicity (and computation), we consider m l–tuples at each layer, selected randomly from all possible
v For a new test point x, we use the sign patterns across all layers for classification
![Page 27: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/27.jpg)
Using all layersHow to integrate the info from all layers?
vl : layerv i : index from 1 to m v : the set of l indices indicating which
hyperplanes are selected in the ith l-tuplev t : a possible sign pattern of l +/- 1sv g : a class index (from 1 to G)v Pg|t : the # of training points from the gth class
having sign pattern t from the hyperplanes in
![Page 28: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/28.jpg)
Using all layersv Pg|t : the # of training points from the gth class
having sign pattern t from the hyperplanes in
![Page 29: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/29.jpg)
Using all layersv Pg|t : the # of training points from the gth class
having sign pattern t from the hyperplanes in
fraction of training points in classg out of all points with pattern t
![Page 30: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/30.jpg)
Using all layersv Pg|t : the # of training points from the gth class
having sign pattern t from the hyperplanes in
fraction of training points in classg out of all points with pattern t
gives more weight to group g when its size is much different than others with same sign pattern
![Page 31: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/31.jpg)
Our method: training
v “For each layer, pick the l–tuples and then compute all values of r(l,i,t,g)”
![Page 32: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/32.jpg)
Our method: testing
v “For a new point x with sign pattern t*, compute the sum of all r(l,i,t*,g) for each class g and then assign the label g which has the largest sum.”
![Page 33: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/33.jpg)
Results (L=1)
![Page 34: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/34.jpg)
Results (L=4)
![Page 35: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/35.jpg)
Results (L=5)
![Page 36: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/36.jpg)
Results
![Page 37: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/37.jpg)
Results
![Page 38: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/38.jpg)
Results (L=1)
![Page 39: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/39.jpg)
Results (L=4)
![Page 40: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/40.jpg)
Results (L=5)
![Page 41: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/41.jpg)
Results (L=5)
![Page 42: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/42.jpg)
Theory
![Page 43: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/43.jpg)
Theory
Ag|t : the angle of class g with sign pattern t for the ith l –tuple in layer l
![Page 44: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/44.jpg)
Theory
![Page 45: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/45.jpg)
Theory
vAs m à ∞, P(correct label) à 1
![Page 46: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/46.jpg)
Theory
![Page 47: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/47.jpg)
Take-awayv Simple classification from binary data
v Efficient storage of the datav Efficient and simple algorithmv Theoretical analysis possiblev Already competes with state of the art
v Future workv Dithers to allow for more complicated
geometries?v Theoretical analysis of the discrete case?
![Page 48: Simple Classification using Binary Datadeanna/simpleclass.pdf · 2017-10-09 · Option 1 : Build bigger computing systems vWe need the resources vFundamental limitations vWasteful](https://reader033.vdocument.in/reader033/viewer/2022050103/5f420e29ce00f16b2b36b1df/html5/thumbnails/48.jpg)
� math.ucla.edu/~deanna
� “Simple Classification using Binary Data” – Needell, Saab, Woolf