Haim Kermany
Learning Decision Treesusing the Fourier Spectrum
By
Eyal Kushilevitz
Yishay Mansour
( )h x
What is Learning
( )f x( ,?)a
( )f a
What are we Learning?
Boolean Function: :{0,1 1} { ,1}nf
1 2 3 4 5 1 2 3 4 1: ( , , , , ) ( ) ( )ex f x x x x x x x x x x
•Term (and of literals(
1 2 3 4 5 1 2 4 1 2 4: ( , , , , )ex f x x x x x x x x x x x
(01011) (0 1) (0 1 0) 1f
•DNF (or of terms)
1 2 3 4 5 1 2 4 2 3 4 5: ( , , , , )ex f x x x x x x x x x x x x
•DT , : See latter
Randomized Algorithm
( ) ( ) ?h x f x Does always All inputs
( )Prob ( )h x f x We Want:
( ) 1h x
( ) 1f x
( ) ( )h x f xDo we always succeed ?
Prob =
Pr ( )ob Pr ( )ob
fail
h x f x 0 Deterministic Algorithem
0 ( ) ( )h x f x
( ) 1,1sign g x
2E ( ) ( )f x g x
:g approcximates f
approximation
P rob ( ) ( )f x sign g x
( ) and thenif f x boolean g approcximates f
What we want
Fourier Transform
i
i1
1 if mod 2 = 0
1 if mod 2 = 1( ... ) i i
i i
z nx z
x zx x
1 0 ( )1
01110
011010110 =+1
(1+ 1)mod 2=0
0,1
ˆ ( )n
z z
z
f f x f x
{0,1}nz x z
t-phase functiont-phase function – a function that has at most t Fourier coefficient
2 g approximates f
2
then th O approcximates f
1 g t phase functionif that:g
ˆ0,1
ˆ ˆ( ) ( )i
nz ti
z z z zfz
f x f x h x f x
Only big coefficients
Very small
( ) has all coefficient that start with f x
:{0,1}n kf
0,1
ˆ( )n k
f x f x
{0,1}k
( ) ( ) , is the empty stringf x f x
all coefficient in ( ) = +f x1all coefficient in ( ) f x
0all coefficient in ( ) f x
( )f x
Tree f x
1f x
11f x
0f x
10f x 00f x 01f x
1f x
1f x
n
f x
2
n
What we need
Finding only big coefficients
ˆif f if output n
else 0 ; 1 ;Coef Coef
Coef
Analyzing coef()
for any , 0,1 :k
k 2 2ˆif then f E f
{0,1}
1[ ] ( ) ( )
2 nnx
E fg f x g x
Changing coef()
2if B if output n
else 0 ; 1 ;Coef Coef
Coef 2B E f
Approximating 2E f x
{0,1}
( )kyf x E f xy y
( )f x
2, , ( ) ( )x y zE f x E f xy f xz y z
2
{0,1}
2
{0,1}
, ,
, ,
( )
( ) ( )
( ) ( )
( ) ( )
( ) ( )
n k
k
x
x y
x y y
x y z
x y z
x y z
E f x
E E f xy y
E E f xy y E f xy y
E E f xy y E f xz z
E f xy y f xz z
E f xy f xz y z
Proof :
Approx do times:m
choose 0,1 ;k
iy choose 0,1 ;
n k
ix
1
Let ;i i i i i ii m
B AVG f y x f y z y z
Approximating 2E f x
2
approximating
E f x
MQ
choose 0,1 ;k
iy
Changing coef()
if output n
else 0 ; 1 ;Coef Coef
Coef
2f 2i B
B Approx Save Side
Finding
m 22 1
4 222Pr 4m
E f B e
41 1logm O
with probability 1 : 22 2 2E f B
2 22 4 2E f B
22 2
4E f
22 2 2E f B
Coef() output
22 2ˆ 2ˆ will be output
z z z
z
z f E f B
f
ˆevery will be outputzf
Coaf() time
2 22
1 f E f
for any :k
2 22 4 2There wont be a recursive call
E f B
222
4 4nE f
22
2 2
2
1 f E f
for any :k
2
22
4 4f E f
for any :k
24There will be at most recursive call
(every call work )
n
m
There will be
coefficients
1( )poly
Running time is
1 1( , , )poly n
Finding
h x
ˆoutput z zz Z
h x f
Z Coef
ˆ approximate zz Z f
find h x
ˆ [ ]z zf E f
Conclusion
2 g approximates f 1 g t phase function
if that:g
1 1
Then there is exist a
that output a function , such that with
probability 1 .
The algorithm run in ( , , , )
randomized algorithm
h
O approximates f
poly n t log
ˆ0,1 ( )1
ˆ ˆ( ) ( )i
nzi
z z z zfz L f
f x f x h x f x
f h approximates f
1ˆ( ) zz
L f f
1( )L fHow to find h(x)
use Coaf() again
Best algorithm ever
1 11
There is a
that for boolean function
output a function , such that with
probability 1 .
The algorithm run in
any
( , , , )
randomized algorithm
f
h
h approximates f
poly n L f log
1L f
Can be exp(n) 1L f
Decision Tree (DT)
+1-1
-1
-1-1-1 +1
+1
+1
+1
+1
01011
11100
10101
11001
10010
00010
0001010010 11100
11100
1 ( )f DT L f m
the of nodes in the treem number
Decision Tree (DT)
Proof:
1 1
There is a
that for any output a function ,
such that with probability 1
.
The algorithm run in ( , , , )
randomized algorithm
f DT h
h approximates f
poly n m log
Exacting
depth of the tree f fd T T
ˆ ˆ
2 fz z d T
kf k f
ˆ ˆz z zf g
:g approcximates f
choose:
21 12 2d
1 1
2 2 fd T ˆ zround g
Final Result
1
There is a
that for any
output a function , such that
with probability 1
.
The algor
ithm run in ( , , log )
randomized algorithm
f DT
h
poly n m
h f
