cube pruning as heuristic search
DESCRIPTION
Cube Pruning as Heuristic Search. Mark Hopkins and Greg Langmead Language Weaver, Inc. Motivation: Speed vs Accuracy. Accuracy (e.g., BLEU) is very important But time is money for the customer: throughput for LW: cpu time on servers for SaaS solution - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/1.jpg)
Cube Pruning asHeuristic Search
Mark Hopkins and Greg Langmead
Language Weaver, Inc.
![Page 2: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/2.jpg)
04/22/23 6
• Accuracy (e.g., BLEU) is very important• But time is money– for the customer: throughput– for LW: cpu time on servers for SaaS solution
• Our customers expect 1000-3000 words per minute in one thread– and linear speedup with multiple threads
• That’s 0.3-1.0 seconds per sentence• Can syntax be viable at product speeds?
Motivation: Speed vs Accuracy
![Page 3: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/3.jpg)
04/22/23 7
il ne va pas
0 1 2 3 4
[0,1] item1[0,1] item2 [0,1] item1[0,1] item2
[1,2] item1[1,2] item2[1,2] item3
[1,2] item1[1,2] item2[1,2] item3
[2,3] item1[2,3] item1 [3,4] item1[3,4] item2[3,4] item3
[3,4] item1[3,4] item2[3,4] item3
[0,2] item1[0,2] item2[0,2] item1[0,2] item2
[2,4] item1[2,4] item2[2,4] item3
[2,4] item1[2,4] item2[2,4] item3
[1,3] item1[1,3] item2[1,3] item1[1,3] item2
[0,3] item1[0,3] item2[0,3] item3
[0,3] item1[0,3] item2[0,3] item3
[1,4] item1[1,4] item2[1,4] item1[1,4] item2
Cube pruning targets CKY
decoding
![Page 4: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/4.jpg)
04/22/23 8
il ne va pas
0 1 2 3 4
[1,4] item1[1,4] item2[1,4] item1
[1,4] item2
an item encodes a (set of) translation(s) of a partial sentence
![Page 5: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/5.jpg)
04/22/23 9
How are items
created?
2 3 4 5
[4,5] item1[4,5] item2 [4,5] item3
[4,5] item1[4,5] item2 [4,5] item3
[2,3] item1[2,3] item2[2,3] item3
[2,3] item1[2,3] item2[2,3] item3
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4
????????
![Page 6: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/6.jpg)
04/22/23 10
Items are created by combining items
from complementary
spans.
2 3 4 5
[4,5] item1[4,5] item2 [4,5] item3
[4,5] item1[4,5] item2 [4,5] item3
[2,3] item1[2,3] item2[2,3] item3
[2,3] item1[2,3] item2[2,3] item3
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4[2,4] item3
[4,5] item2
[2,5] item1
![Page 7: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/7.jpg)
04/22/23 11
[ 2,4 , NP , the*car ]
an item consists of three parts:
span postcondition
carry
What is an item?
![Page 8: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/8.jpg)
04/22/23 12
an item consists of three parts:
span postcondition
carry
[ 2,4 , NP , the*car ]
What is an item?
![Page 9: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/9.jpg)
04/22/23 13
an item consists of three parts:
span postcondition
carry
What is an item?
[ 2,4 , NP , the*car ]
![Page 10: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/10.jpg)
04/22/23 14
CKY Item Creation
[2,3,A,a*b]
[3,5,B,b*a]
B < A0 B1 , B1 A0 >
[2,5,B,b*b]
postcondition carry
preconditions
postconditioncost( subitem1)
cost( subitem2)
cost( rule )
+ interaction( rule, subitems)
cost( new item )
![Page 11: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/11.jpg)
04/22/23 15
The Item Creation Problem
2 3 4 5
[4,5] item1[4,5] item2 [4,5] item3
[4,5] item1[4,5] item2 [4,5] item3
[2,3] item1[2,3] item2[2,3] item3
[2,3] item1[2,3] item2[2,3] item3
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[3,5] item1[3,5] item2[3,5] item3[3,5] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4
[2,4] item1[2,4] item2[2,4] item3[2,4] item4
????????even if we just store
1000 items here
…and we have only 1000s of grammar
rules
…and 1000 items here
1000 1000 10001000000000…then item creation can still
take a very long time
![Page 12: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/12.jpg)
04/22/23 16
The Item Creation Problem
2 3 4 5
10001000
10001000 10001000
10001000
????????
is there a better way to enumerate the 1000 items of
lowest cost for this span without going through the millions of candidate items and taking the best 1000?
this is the problem
that cube pruning
addresses
![Page 13: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/13.jpg)
1704/22/23
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[?,?,?,?]
[?,?,?,?]? < ?0 ?1 , ? ? >
[?,?,?,?]
So far we have:
A demonstration of incremental
CKY item creation for span [2,5]
![Page 14: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/14.jpg)
18
[2,4] [4,5] [2,3] [3,5]
04/22/23
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,?,?]
[3,5,?,?]? < ?0 ?1 , ? ? >
[2,5,?,?]
So far we have:
![Page 15: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/15.jpg)
19
[2,4] [4,5] [2,3] [3,5]
A B
04/22/23
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,A,?]
[3,5,?,?]? < A0 ?1 , ? ? >
[2,5,?,?]
So far we have:
![Page 16: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/16.jpg)
04/22/23 20
[2,4] [4,5] [2,3] [3,5]
A B
A B
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,A,?]
[3,5,B,?]? < A0 B1 , ? ? >
[2,5,?,?]
So far we have:
![Page 17: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/17.jpg)
04/22/23 21
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
accept rule(A,B,1)?
rule(A,B,k) is the kth lowest cost rule whose
preconditions are <A,B>
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,A,?]
[3,5,B,?]B < A0 B1 , B1
A0 >[2,5,B,?]
So far we have:
![Page 18: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/18.jpg)
04/22/23 22
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y
accept rule(A,B,1)?
accept item(2,3,A,1)?
item(2,3,A,k) is the kth lowest cost item of span
[2,3] whose postcondition is A
n
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,A,a*b][3,5,B,?]
B < A0 B1 , B1 A0 >
[2,5,B,?*b]
So far we have:
![Page 19: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/19.jpg)
04/22/23 23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,1)?y n
accept item(3,5,B,2)?
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1 A0 >
[2,5,B,b*b]
We want:
[2,3,A,a*b][3,5,B,b*a]B < A0 B1 , B1
A0 >[2,5,B,b*b
]
So far we have:
![Page 20: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/20.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,1)?y n
accept item(3,5,B,2)?
this is a search space
The Item Creation Problem,
rephrased:find the n lowest-cost goal nodes of this search
space 24
![Page 21: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/21.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
so if we can come up with lower-bounds on the
best-cost reachable goal
node from here……and here
…and here
…then we can just run A* on this
search space to find the n goal nodes of lowest
cost (without searching the entire space)
25
![Page 22: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/22.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
h = -infinity
h = -infinity
h = greedy lookahead cost
h = greedy lookahead cost
h = greedy lookahead cost
26
![Page 23: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/23.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
AB
h = -infinity
h = -infinity
27
accept rule(A,B,1)?
y naccept item(3,5,B,1)?
y n
accept item(2,3,A,1)?ny
h =
[2,5,B,b*b]cost = 7.48
7.48
![Page 24: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/24.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
AB
h = -infinity
h = -infinity
28
accept rule(A,B,1)?
y naccept item(3,5,B,1)?
y
accept item(2,3,A,1)?ny
h = 7.48
accept item(2,3,A,2)?
y n
y naccept item(3,5,B,1)?
[2,5,A,a*b]cost = 12.26
h =
12.26
![Page 25: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/25.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
AB
h = -infinity
h = -infinity
29
accept rule(A,B,1)?
y naccept item(3,5,B,1)?
y
accept item(2,3,A,1)?ny
h = 7.48
accept item(2,3,A,2)?
y n
y naccept item(3,5,B,1)?
[2,5,A,a*b]cost = 12.26
h = 12.26
not a lower bound
[2,5,A,a*b]cost = 5.42
7.48
<
![Page 26: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/26.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
h = -infinity
h = -infinity
h = greedy lookahead cost
h = greedy lookahead
cost
h = greedy lookahead cost
30
admissible
not admissibl
e
therefore A* will not find the n best
solutions, it will only find
n good solutions
![Page 27: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/27.jpg)
04/22/23 31
A*search
Cube pruning
vs.
![Page 28: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/28.jpg)
[2,4][4,5]
04/22/23 32
A*search
Cube pruning
Cube pruning begins by forming
cubes
[2,3] [3,5]
for each choice of subspans
and preconditions
[2,3] A
[3,5] B
[2,3] A
[3,5] A
[2,4] A
[4,5] B
[2,4] A
[4,5] A
[2,3] B
[3,5] A
[2,3] B
[3,5] B
[2,4] B
[4,5] A
[2,4] B
[4,5] B
![Page 29: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/29.jpg)
h = -infh = -inf
04/22/23 33
A*search
Cube pruning
Cube pruning begins by forming
cubes
[2,3] B
[3,5] A
[2,3] A
[3,5] B
A* search visits nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
h = -inf
-inf -inf -inf -inf
therefore, it will begin by visiting
all nodes with –inf heuristics
[2,3] A
[3,5] A
[2,3] B
[3,5] B
[2,4] A
[4,5] B
[2,4] A
[4,5] A
[2,4] B
[4,5] A
[2,4] B
[4,5] B
![Page 30: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/30.jpg)
04/22/23 34
A*search
Cube pruning
Cube pruning begins by forming
cubes
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3] A
[3,5] B
[2,3] B
[3,5] A
[2,3] B
[3,5] B
[2,3] A
[3,5] A
[2,4] A
[4,5] B
[2,4] B
[4,5] A
[2,4] A
[4,5] A
[2,4] B
[4,5] B
![Page 31: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/31.jpg)
04/22/23 35
A*search
Cube pruning
What is a cube?A* search visits
nodes in order of increasing
heuristic value[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3] A
[3,5] B
a cube is a set of three axes
![Page 32: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/32.jpg)
04/22/23 36
A*search
Cube pruning
What is a cube?A* search visits
nodes in order of increasing
heuristic value[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3]A
[3,5]B
item(2,3,A,1)item(2,3,A,2)item(2,3,A,3)
item(3,5,B,3)item(3,5,B,2)item(3,5,B,1)
rule(A,B,1)
rule(A,B,2)
rule(A,B,3)
rule(A,B,4)
[2,3] A
[3,5] B
[2,3] A
[3,5] B
sorted by increasing cost
[2,3] A
[3,5] B
sorted by increasing cost
sorted by increasing cost
[2,3] A
[3,5] B
![Page 33: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/33.jpg)
04/22/23 37
A*search
Cube pruning
Thus each choice of object from
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3]A
[3,5]B
item(2,3,A,1)item(2,3,A,2)item(2,3,A,3)
item(3,5,B,3)item(3,5,B,2)item(3,5,B,1)
rule(A,B,1)
rule(A,B,2)
rule(A,B,3)
rule(A,B,4)
here
item(2,3,A,2)
here
item(3,5,B,1)
and here
rule(A,B,4)
![Page 34: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/34.jpg)
04/22/23 38
A*search
Cube pruning
…creates a new item for [2,5]
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3]A
[3,5]B
item(2,3,A,1)item(2,3,A,2)item(2,3,A,3)
item(3,5,B,3)item(3,5,B,2)item(3,5,B,1)
rule(A,B,1)
rule(A,B,2)
rule(A,B,3)
rule(A,B,4)
item(2,3,A,2)
item(3,5,B,1)
rule(A,B,4)
[2,3,A,a*b]
[3,5,B,b*a]
B < A0 B1 , B1 A0 >
[2,5,B,b*b]
![Page 35: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/35.jpg)
04/22/23 39
A*search
Cube pruning
If we take the best representative from each axis (i.e. the “1-1-1”)…
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3]A
[3,5]B
item(2,3,A,1)item(2,3,A,2)item(2,3,A,3)
item(3,5,B,3)item(3,5,B,2)item(3,5,B,1)
rule(A,B,1)
rule(A,B,2)
rule(A,B,3)
rule(A,B,4)
…then we expect the
resulting item to have a low
cost,since:
cost( new item ) =cost( subitem1 )
+ cost( subitem2
)+
cost( rule )+ interaction( rule, subitems )
![Page 36: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/36.jpg)
04/22/23 40
A*search
Cube pruning
Though we are not guaranteed this, because:
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B
[2,3]A
[3,5]B
item(2,3,A,1)item(2,3,A,2)item(2,3,A,3)
item(3,5,B,3)item(3,5,B,2)item(3,5,B,1)
rule(A,B,1)
rule(A,B,2)
rule(A,B,3)
rule(A,B,4)
cost( new item ) =cost( subitem1 )
+ cost( subitem2
)+
cost( rule )+ interaction( rule, subitems )
this cost is not monotonic
![Page 37: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/37.jpg)
04/22/23 41
A*search
Cube pruning
A* search visits nodes in order of
increasing heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
Cube pruning proceeds by
creating the 1-1-1 item of every cube
2.4 4.6 7.9 3.2
5.5 9.2 6.2 4.4
and scoring them
![Page 38: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/38.jpg)
04/22/23 42
A*search
Cube pruning
Meanwhile, A* search has scored its frontier nodes
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
Cube pruning proceeds by
creating the 1-1-1 item of every cube
and scoring them
accept rule(A,B,1)
accept item(2,3,A,1)
accept item(3,5,B,1)
111
2.4
2.4 4.6 7.9 3.2
5.5 9.2 6.2 4.4
![Page 39: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/39.jpg)
04/22/23 43
A*search
Cube pruning
Meanwhile, A* search has scored its frontier nodes
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
Cube pruning proceeds by
creating the 1-1-1 item of every cube
2.4 4.6 7.9 3.2
5.5 9.2 6.2 4.4
and scoring them
2.44.64.46.23.2 7.9 5.5 9.2
![Page 40: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/40.jpg)
04/22/23 44
A*search
Cube pruning
Meanwhile, A* search has scored its frontier nodes
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
At this point, cube pruning takes the best item it has
created.
2.4 4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
2.44.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
![Page 41: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/41.jpg)
04/22/23 45
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
2.4
![Page 42: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/42.jpg)
04/22/23 46
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
211
1112.4 4.8
![Page 43: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/43.jpg)
04/22/23 47
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
2.4 4.8
![Page 44: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/44.jpg)
04/22/23 48
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
4.8
y n121
1112.4 7.1
![Page 45: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/45.jpg)
04/22/23 49
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
4.8
y n2.4 7.1
![Page 46: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/46.jpg)
04/22/23 50
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
4.8
y n7.1
y n112
1112.4 3.6
![Page 47: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/47.jpg)
04/22/23 51
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
4.8
y n7.1
y n2.4 3.6
this is a goal node
![Page 48: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/48.jpg)
04/22/23 52
A*search
Cube pruning
A* search continues to visit nodes in
increasing order of heuristic value
[2,4] [4,5] [2,3] [3,5]
A B A B
A B A B A B A B111
111
111
111
111
111
111
111
4.6 7.9 3.2
5.5 9.2 6.2 4.4
It keeps this item.
4.64.46.23.2 7.9 5.5 9.2
KEPT
And generates its
“one-off” items.
211
121
112
4.8
7.1 3.6
y n
y n
y n
4.8
7.1
3.6
this is a goal node
111
So A* keeps this item.
KEPT
![Page 49: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/49.jpg)
04/22/23 53
A*search
Cube pruning
=alm
ost
(+ node tying)
=
![Page 50: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/50.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
h = -infinity
h = -infinity
h = greedy lookahead cost
h = greedy lookahead cost
h = greedy lookahead cost
54
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,2)?
accept item(3,5,B,1)?
?
Cube pruning was specifically designed for
hierarchical phrase-based MT
which uses only a small number of distinct
postconditions
But say our use case was string-to-tree MT,
in the style of (Galley et al 2006)
![Page 51: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/51.jpg)
04/22/23 55
Average number of search nodes
visited, per sentence
Nodes by Type Cube Pruning
subspan 12936
precondition 851458
rule 33734
item 119703
goal 74618
TOTAL 1092449
BLEU 38.33
Arabic-English NIST 2008
the early nodes with
infinite heuristics dominate the search
time
![Page 52: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/52.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
h = -infinity
h = -infinity
h = greedy lookahead cost
h = greedy lookahead cost
h = greedy lookahead cost
56
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,2)?
accept item(3,5,B,1)?
?h = something better
h = something better
!
![Page 53: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/53.jpg)
Nodes by Type Cube Pruning Augmented CP
subspan 12936 12792
precondition 851458 379954
rule 33734 33331
item 119703 118889
goal 74618 74159
TOTAL 1092449 619125
BLEU 38.33 38.22
04/22/23 57
Number of search nodes
visitedNodes by Type Cube Pruning
subspan 12936
precondition 851458
rule 33734
item 119703
goal 74618
TOTAL 1092449
BLEU 38.33
Arabic-English NIST 2008
![Page 54: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/54.jpg)
04/22/23 58
Tradeoff curves Arabic-English NIST 2008
![Page 55: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/55.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
y n
h = -infinity
h = -infinity
h = greedy lookahead cost
h = greedy lookahead cost
h = greedy lookahead cost
59
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,2)?
accept item(3,5,B,1)?
h = admissible
h = admissible
h = admissible
h = admissible
h = admissible
Cube pruning becomes exact.
We found that our exact version of cube pruning
had a similar time/quality curve to the original
inexact version of cube pruning.
However it was not as effective as our
“augmented” version of cube pruning.
This is all interesting, but somewhat beside the
point.
![Page 56: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/56.jpg)
04/22/23 60
What
does cube pruning teach us?
esson
![Page 57: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/57.jpg)
04/22/23
[2,4] [4,5] [2,3] [3,5]
A B
A B
y n
y n
y n
accept rule(A,B,1)?
accept item(2,3,A,1)?
accept item(3,5,B,1)?y n
accept item(3,5,B,2)?
this is a search space
61
It tells us that it is useful to
frame the CYK Item
Generation Problem as a
heuristic search problem.
Once this realization is
made, we suddenly have
many more avenues
available to us, when
implementing a CKY decoder for a particular use
case
![Page 58: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/58.jpg)
04/22/23 62
h = -infinity
h = greedy lookahead cost
h = greedy lookahead cost
h = greedy lookahead cost
h = -infinity
h = something better
h = something better
We can change the heuristics.
![Page 59: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/59.jpg)
04/22/23 63
We can change the search algorithm.
We can change the heuristics.
![Page 60: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/60.jpg)
04/22/23 64
We can change the search algorithm.
For instance, instead of A*…
![Page 61: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/61.jpg)
04/22/23 65
We can change the search algorithm.
For instance, instead of A*…
…we could try a depth-first strategy like depth-first branch-and-bound, and take advantage of its anytime properties.
![Page 62: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/62.jpg)
04/22/23 66
We can change the search algorithm.
We can change the search space.
![Page 63: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/63.jpg)
04/22/23 67
What
does cube pruning teach us?
esson
We end up with a speedup technique
which is:simplegeneral
well-studiedeasily adaptable to
new use cases
![Page 64: Cube Pruning as Heuristic Search](https://reader036.vdocument.in/reader036/viewer/2022070418/56815809550346895dc57880/html5/thumbnails/64.jpg)
04/22/23 68
Thank you.
Questions?