generalized and heuristic-free feature construction for improved accuracy wei fan ‡, erheng zhong...
Post on 22-Dec-2015
222 Views
Preview:
TRANSCRIPT
Generalized and Heuristic-Free Feature Construction
for Improved Accuracy
Wei Fan‡, Erheng Zhong†, Jing Peng*, Olivier Verscheure‡, Kun Zhang§, Jiangtao Ren†,
Rong Yan‡ and Qiang Yang¶
‡IBM T. J. Watson Research Center†Sun Yat-Sen University
*Montclair State University §Xavier University of Lousiana
Facebook, Inc¶Hong Kong University of Science and Technology
• Construction works when the original pool is not good enough (feature selection won’t work)• Too many choices to construct• Evaluate on local space not always on all the data points• Better Automated
Feature Construction -- Example
3 1 2F F F
XOR like problemNot linearly separable:use both features to construct a “cross” model
Linearly separable:one feature F3 is enough
Main Challenges
To address these, we have 3 main steps
1. Too many ways to construct new features: xy, x-y,x/y, etc• Divide and Conquer
2. Insignificant on the whole data set - highly discriminant in local region • Local Feature Construction and Evaluation
3. Automated – not based on domain knowledge• Automatically adjusted weighting rules
4 binary operators, 1000 original features up to
constructed features64 10
F2 not very usefulunless consideredwith F1
Divide-Conquer
Local Feature Construction and Evaluation
Stopping Criteria: 1.The number of instances in the node is smaller than a threshold2.The node only contains examples from one class
ConstructedFeatures(org + new)
Every node …
(1)
F
(3)
(4)
Weighted
1. Random subset of orig features
2. “Weighted random” subset of operators
(2)
Weighting Rule
Weighting Rule
• Weight is proportional to the info-gain of features constructed by the operator.
Sum of its past info gain
Properties
• Number of features is bounded.
• Highly weighted operator is expected to perform better in its two child nodes (see paper)
• FCTree’s error is bounded.
– also explains why the features are of high quality
Experiment – Data Set
• UCI repository (Balanced)• Caltech-256 database: An image database of 256 obje
ct categories. Each category is processed via a 177-dimensional color correlogram (Balanced)
• Landmine collection: Collected via remote sensing techniques (Skewed)
• Nuclear Ban data source: A nuclear explosion detection problem used by ICDM’08 contest (Skewed)
Experiment -- Baseline methods
• Original Features• TFC:
– enumerates all possible features generated by operators
• NB,SVM and C45• Operators
• FCTree:
Performance--Blannced Data
Best in 23 out of 33 comparisions
Performance--Skew Data
Best in 25 out of 33 comparisions
Scalability Analysis
Strength of Weighting Rule
Original
FCTree
177 dimensioncolor correlogram
Conclusion
• Key points– Divide-conquer to avoid exhaustive enu
meration;– Local feature construction subspace
evaluation– Weighting rules based search: domain
knowledge free and provable performance.
• Code and data available from the authors
top related