foundations of constraint processing, fall 2005 september 11, 2005backtracking1 foundations of...
Post on 19-Dec-2015
218 views
TRANSCRIPT
September 11, 2005 Backtracking 1
Foundations of Constraint Processing, Fall 2005
Foundations of Constraint Processing
CSCE421/821, Fall 2005: www.cse.unl.edu/~choueiry/F05-421-821/
Berthe Y. Choueiry (Shu-we-ri)
Avery Hall, Room 123B
Tel: +1(402)472-5444
Intelligent Backtracking Algorithms
September 11, 2005 Backtracking 2
Foundations of Constraint Processing, Fall 2005
Reading• Required reading
Hybrid Algorithms for the Constraint Satisfaction Problem [Prosser, CI 93]
• Recommended reading– Chapters 5 and 6 of Dechter’s book– Tsang, Chapter 5
• Notes available upon demand– Notes of Fahiem Bacchus: Chapter 2, Section 2.4– Handout 4 and 5 of Pandu Nayak (Stanford)
September 11, 2005 Backtracking 3
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search
• Hybrid backtracking algorithms
• Looking ahead
• Evaluation of (deterministic) BT search algorithms
September 11, 2005 Backtracking 4
Foundations of Constraint Processing, Fall 2005
Backtrack search (BT)
Var 1 v1 v2
S
• Variable/value ordering
• Variable instantiation
• (Current) path
• Current variable
• Past variables
• Future variables
• Shallow/deep levels /nodes
• Search space / search tree
• Back-checking
• Backtracking
September 11, 2005 Backtracking 5
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search • Hybrid backtracking algorithms
– Vanilla: BT– Improving back steps: {BJ, CBJ} – Improving forward step: {BM, FC}
• Looking ahead• Evaluation of (deterministic) BT search
algorithms
September 11, 2005 Backtracking 6
Foundations of Constraint Processing, Fall 2005
Two main mechanisms in BT
1. Backtracking: • To recover from dead-ends • To go back
2. Consistency checking: • To expand consistent paths• To move forward
September 11, 2005 Backtracking 7
Foundations of Constraint Processing, Fall 2005
Backtracking
To recover from dead-ends
1. Chronological (BT)
2. Intelligent• Backjumping (BJ)• Conflict directed backjumping (CBJ)• With learning algorithms (Dechter Chapt 6.4)• Etc.
September 11, 2005 Backtracking 8
Foundations of Constraint Processing, Fall 2005
Consistency checking
To expand consistent paths1. Back-checking: against past variables
• Backmarking (BM)
2. Look-ahead: against future variables• Forward checking (FC) (partial look-ahead)• Directional Arc-Consistency (DAC) (partial
look-ahead)• Maintaining Arc-Consistency (MAC) (full
look-ahead)
September 11, 2005 Backtracking 9
Foundations of Constraint Processing, Fall 2005
Hybrid algorithms
Backtracking + checking = new hybrids
BT BJ CBJ
BM BMJ BM-CBJ
FC FC-BJ FC-CBJ
Evaluation:
• Empirical: Prosser 93. 450 instances of Zebra
• Theoretical: Kondrak & Van Beek 95
September 11, 2005 Backtracking 10
Foundations of Constraint Processing, Fall 2005
Notations (in Prosser’s paper)
• Variables: Vi, i in [1, n]• Domain: Di = {vi1, vi2, …,viMi}• Constraint between Vi and Vj: Ci,j
• Constraint graph: G• Arcs of G: Arc(G)• Instantiation order (static or dynamic)• Language primitives: list, push, pushnew,
remove, set-difference, union, max-list
September 11, 2005 Backtracking 11
Foundations of Constraint Processing, Fall 2005
Main data structures• v: a (1xn) array to store assignments
– v[i] gives the value assigned to ith variable – v[0]: pseudo variable (root of tree), backtracking to
v[0] indicates insolvability
• domain[i]: a (1xn) array to store the original domains of variables
• current-domain[i]: a (1xn) array to store the current domains of variables– Upon backtracking, current-domain[i] of future
variables must be refreshed
• check(i,j): a function that checks whether the values assigned to v[i] and v[j] are consistent
September 11, 2005 Backtracking 12
Foundations of Constraint Processing, Fall 2005
Generic search: bcssp1. Procedure bcssp (n, status)2. Begin3. consistent true4. status unknown5. i 16. While status = unknown
7. Do Begin8. If consistent
9. Then i label (i, consistent)10. Else i unlabel (i, consistent)
11. If i > n12. Then status “solution”13. Else If i=0 then status “impossible”14. End
15. End
•Forward move: x-label
•Backward move: x-unlabel
•Parameters: current variable, Boolean
•Return: new current variable
September 11, 2005 Backtracking 13
Foundations of Constraint Processing, Fall 2005
Chronological backtracking (BT)
• Uses bt-label and bt-unlabel• When v[i] is assigned a value from current-domain[i], we
perform back-checking against past variables (check(i,k))
• If back-checking succeeds, bt-label returns i+1• If back-checking fails, we remove the assigned value
from current-domain[i], assign the next value in current-domain[i], etc.
• If no other value exists, v[i-1] is un-instantiated and we seek a new value for it… (notation: in general v[h])
• For all future variables j: current-domain[j] = domain[j]• For all past variables g: current-domain[g] domain[g]
September 11, 2005 Backtracking 14
Foundations of Constraint Processing, Fall 2005
BT-label1. Function bt-label(i,consistent): INTEGER2. BEGIN3. consistent false4. For v[i] each element of current-domain[i] while not consistent
5. Do Begin6. consistent true7. For h 1 to (i-1) While consistent8. Do consistent check(i,h)9. If not consistent10. Then current-domain[i] remove(v[i], current-domain[i])11. End
12. If consistent then return(i+1) ELSE return(i)13. END
Terminates:
• consistent=true, return i+1
• consistent=false, current-domain[i]=nil, returns i
September 11, 2005 Backtracking 15
Foundations of Constraint Processing, Fall 2005
BT-unlabel1. FUNCTION bt-unlabel(i,consistent):INTEGER
2. BEGIN
3. h i -1
4. current-domain[i] domain[i]
5. current-domain[h] remove(v[h],current-domain[h])
6. consistent current-domain[h] nil
7. return(h)
8. END • Is called when consistent=false and current-domain[i]=nil
• Selects vh to backtrack to
• Uninstantiates all variables between vh and vi
• Removes v[h] from current-domain [h]
• Sets consistent to true if current-domain[h] 0
• Returns h
September 11, 2005 Backtracking 16
Foundations of Constraint Processing, Fall 2005
Example: BT (the dumbest example ever)
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
V2
V1
V3
V4
V5
CV3,V4={(V3=1,V4=3)}
CV2,V5={(V2=5,V5=1),(V2=5,V5=4)}
-
v[1]
v[2]
v[3]
v[4]
v[5]
v[0]
1
1
1
1
21 3 4
2 3 4 5
etc…
September 11, 2005 Backtracking 17
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search • Hybrid backtracking algorithms
– Vanilla: BT– Improving back steps: BJ, CBJ – Improving forward step: BM, FC
• Looking ahead• Evaluation of (deterministic) BT search
algorithms
September 11, 2005 Backtracking 18
Foundations of Constraint Processing, Fall 2005
Danger of BT: thrashing• BT assumes that the instantiation of v[i]
was prevented by a bad choice at (i-1). • It tries to change the assignment of v[i-1]• When this assumption is wrong, we suffer
from thrashing (exploring ‘barren’ parts of solution space)
• Backjumping (BT) tries to avoid that– Jumps to the reason of failure – Then proceeds as BT
September 11, 2005 Backtracking 19
Foundations of Constraint Processing, Fall 2005
Backjumping (BJ)• Tries to reduce thrashing by saving some
backtracking effort• When v[i] is instantiated, BJ remembers
v[h], the deepest node of past variables that v[i] has checked against.
• Uses: max-check[i], global, initialized to 0• At level i, when check(i,h) succeeds
max-check[i] max(max-check[i], h)
• If current-domain[h] is getting empty, simple chronological backtracking is performed from h– BJ jumps then steps!
12
3
0
23
1
i
h-1h-1
h
h
h-2
0
0
0
Current variablePast variable
September 11, 2005 Backtracking 20
Foundations of Constraint Processing, Fall 2005
BJ: label/unlabel• bj-label: same as bt-label, but updates max-
check[i]• bj-unlabel, same as bt-unlabel but
– Backtracks to h = max-check[i]– Resets max-check[j] 0 for j in [h+1,i]
Important: max-check is the deepest level we checked against, could have been success or could have been failure
12
3
0
23
1
i
h-1h-1
h
h
h-2
0
0
0
September 11, 2005 Backtracking 21
Foundations of Constraint Processing, Fall 2005
Example: BJ
2
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
V2
V1
V3
V4
V5
CV2,V4={(V2=1,V4=3)}
CV1,V5={(V1=1,V5=2)}
CV2,V5={(V2=5,V5=1)}
-
v[1]
v[2]
v[3]
v[4]
v[5]
v[0] = 0
1
1
1
1
21 3 4
2 3 4 5
Max-check[1] = 0
Max-check[2] = 1
max-check[4] = 3
max-check[5] = 2
V4=1, fails for V2, mc=2V4=2, fails for V2, mc=2 V4=3, succeeds
V5=1, fails for V1, mc=1V5=2, fails for V2, mc=2V5=3, fails for V1V5=4, fails for V1V5=5, fails for V1
September 11, 2005 Backtracking 22
Foundations of Constraint Processing, Fall 2005
Conflict-directed backjumping (CBJ)
• Backjumping– jumps from v[i] to v[h], – but then, it steps back from v[h] to v[h-1]
• CBJ improves on BJ– Jumps from v[i] to v[h]– And jumps back again, across conflicts
involving both v[i] and v[h]– To maintain completeness, we jump back to
the level of deepest conflict
September 11, 2005 Backtracking 23
Foundations of Constraint Processing, Fall 2005
CBJ: data structure
• Maintains a conflict set: conf-set• conf-set[i] are first initialized to {0}• At any point, conf-set[i] is a subset of
past variables that are in conflict with i
{0}
{0}
{0}
{0}{0}{0}
conf-set[g]
conf-set[h]
conf-set[i]
01
2
g
h-1
h
i
conf-set
September 11, 2005 Backtracking 24
Foundations of Constraint Processing, Fall 2005
CBJ: conflict-set
{x}
{3}
{1, g, h}
{0}{0}{0}
conf-set[g]
conf-set[h]
conf-set[i]
12
3
g
h-1h
Current variable i
Pas
t var
iabl
es
{3,1, g}
{x, 3,1}
• When a check(i,h) failsconf-set[i] conf-set[i] {h}
• When current-domain[i] empty
1. Jumps to deepest past variable
h in conf-set[i]
2. Updates
conf-set[h] conf-set[h] (conf-set[i] \{h})
• Primitive form of learning (while searching)
September 11, 2005 Backtracking 25
Foundations of Constraint Processing, Fall 2005
Example CBJ{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
{1,2,3,4,5}
V2
V1
V3
V4
V5
{(V1=1, V6=3)}
-
v[1]
v[2]
v[3]
v[4]
v[6]
v[0] = 0
1
1
1
1
21 3
2 3 4 5
conf-set[1] = {0}
conf-set[2] = {0}
conf-set[3] = {0}
{(V4=5, V6=3)}
{(V2=1, V4=3), (V2=4, V4=5)}
conf-set[6] = {1}
{1,2,3,4,5}V6
{(V1=1, V5=3)}
conf-set[4] = {2}
v[5] 21 3
conf-set[6] = {1}conf-set[6] = {1,4}
conf-set[6] = {1,4}conf-set[6] = {1,4}
conf-set[4] = {1, 2}
conf-set[5] = {1}
September 11, 2005 Backtracking 26
Foundations of Constraint Processing, Fall 2005
Backtracking: summary• Chronological backtracking
– Steps back to previous level– No extra data structures required
• Backjumping– Jumps to deepest checked-against variable, then
steps back– Uses array of integers: max-check[i]
• Conflict-directed backjumping– Jumps across deepest conflicting variables– Uses array of sets: conf-set[i]
September 11, 2005 Backtracking 27
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search • Hybrid backtracking algorithms
– Vanilla: BT– Improving back steps: BJ, CBJ – Improving forward step: BM, FC
• Looking ahead• Evaluation of (deterministic) BT search
algorithms
September 11, 2005 Backtracking 28
Foundations of Constraint Processing, Fall 2005
Backmarking: goal
• Tries to reduce amount of consistency checking
• Situation:– v[i] about to be re-assigned k– v[i] k was checked against v[h]g
– v[h] has not been modified
v[h] = g
v[i] kk
September 11, 2005 Backtracking 29
Foundations of Constraint Processing, Fall 2005
BM: motivation• Two situations
1. Either (v[i]=k,v[h]=g) has failed it will fail again2. Or, (v[i]=k,v[h]=g) was founded consistent it will remain consistent
v[h] = g
v[i] kk
v[h] = g
v[i] kk
• In either case, back-checking effort against v[h] can be saved!
September 11, 2005 Backtracking 30
Foundations of Constraint Processing, Fall 2005
Data structures for BM: 2 arrays
0 0 0 0 0 0 0 0 0
0
0
0
0
Num
ber
of v
aria
bles
n
max domain size m
Num
ber
of v
aria
bles
n
• maximum checking level: mcl (n x m)• Minimum backup level: mbl (n x 1)
September 11, 2005 Backtracking 31
Foundations of Constraint Processing, Fall 2005
Maximum checking level
0 0 0 0 0 0 0 0 0
00
0
0
Num
ber
of v
aria
bles
n
max domain size m
• mcl[i,k] stores the deepest variable that v[i]k checked against
• mcl[i,k] is a finer version of max-check[i]
September 11, 2005 Backtracking 32
Foundations of Constraint Processing, Fall 2005
Minimum backup level
Num
ber
of v
aria
bles
n
• mbl[i] gives the shallowest past variable whose value has changed since v[i] was the current variable
• BM (and all its hybrid) do not allow dynamic variable ordering
September 11, 2005 Backtracking 33
Foundations of Constraint Processing, Fall 2005
When mcl[i,k]=mbl[i]=j
v[i] kk
v[j]
mbl[i] = j
BM is aware that• The deepest variable that (v[i] k)
checked against is v[j]• Values of variables in the past of
v[j] (h<j) have not changed
So• We do need to check (v[i] k)
against the values of the variables between v[j] and v[i]
• We do not need to check (v[i] k) against the values of the variables in the past of v[j]
September 11, 2005 Backtracking 34
Foundations of Constraint Processing, Fall 2005
Type a savings
v[h]
v[i] kk
v[j]
mcl[i,k]=h mcl[i,k] < mbl[i]=j
When mcl[i,k] < mbl[i], do not check v[i] k because it will fail
September 11, 2005 Backtracking 35
Foundations of Constraint Processing, Fall 2005
Type b savings
h
v[i] kk
v[j]
v[g]
mcl[i,k]=g
mbl[i] = j
mcl[i,k]mbl[i]
When mcl[i,k] mbl[i], do not check (i,h<j) because they will succeed
September 11, 2005 Backtracking 36
Foundations of Constraint Processing, Fall 2005
Hybrids of BM
• mcl can be used to allow backjumping in BJ
• Mixing BJ & BM yields BMJ– avoids redundant consistency checking (types
a+b savings) and – reduces the number of nodes visited during
search (by jumping)
• Mixing BM & CBJ yields BM-CBJ
September 11, 2005 Backtracking 37
Foundations of Constraint Processing, Fall 2005
Problem of BM and its hybrids: warning
v[m]v[m]
v[g]
v[h]
v[i]
v[g]
v[h]
v[i]
v[m]
v[g]
v[h]
v[i]
v[h]
v[f]
• Backjumping from v[i]:– v[i] backjumps up to v[g]
• Backmarking of v[h]:– When reconsidering v[h], v[h] will
be checked against all f in [m,g( – effort could be saved
• Phenomenon will worsen with CBJ
• Problem fixed by Kondrak & van Beek 95
BMJ enjoys only some of the advantages of BM
Assume: mbl[h] = m and max-check[i]=max(mcl[i,x])=g
September 11, 2005 Backtracking 38
Foundations of Constraint Processing, Fall 2005
Forward checking (FC)• Looking ahead: from current variable, consider all future
variables and clear from their domains the values that are not consistent with current partial solution
• FC makes more work at every instantiation, but will expand fewer nodes
• When FC moves forward, the values in current-domain of future variables are all compatible with past assignment, thus saving backchecking
• FC may “wipe out” the domain of a future variable (aka, domain annihilation) and thus discover conflicts early on. FC then backtracks chronologically
• Goal of FC is to fail early (avoid expanding fruitless subtrees)
September 11, 2005 Backtracking 39
Foundations of Constraint Processing, Fall 2005
FC: data structures
v[i]
v[k]
v[l]
v[n]
v[m]
v[j]
• When v[i] is instantiated, current-domain[j] are filtered for all j connected to i and i<j<n
• reduction[j] store sets of values remove from current-domain[j] by some variable before v[j]
reductions[j] = {{a, b}, {c, d, e}, {f, g, h}}
• future-fc[i]: subset of the future variables that v[i] checks against (redundant)
future-fc[i] = {k, j, n}
• past-fc[i]: past variables that checked against v[j]
• All these sets are treated like stacks
September 11, 2005 Backtracking 40
Foundations of Constraint Processing, Fall 2005
Forward Checking: functions
• check-forward
• undo-reductions
• update-current-domain
• fc-label
• fc-unlabel
September 11, 2005 Backtracking 41
Foundations of Constraint Processing, Fall 2005
FC: functions• check-forward(i,j) is called when instantiating v[i]
– It performs REVISE(j,i)– Returns false if current-domain[j] is empty, true
otherwise– Values removed from current-domain[j] are pushed,
as a set, into reductions[j]
• These values will be popped back if we have to backtrack over v[i] (undo-reductions)
September 11, 2005 Backtracking 42
Foundations of Constraint Processing, Fall 2005
FC: functions• update-current-domain
– current-domain[i] domain[i] \ reductions[i] – actually, we have to iterate over reductions=set of
sets
• fc-label– Attempts to instantiate current-variable– Then filters domains of all future variables (push into
reductions)– Whenever current-domain of a future variable is
wiped-out: • v[i] is un-instantiated and • domain filtering is undone (pop reductions)
September 11, 2005 Backtracking 43
Foundations of Constraint Processing, Fall 2005
Hybrids of FC• FC suffers from thrashing: it is based on BT• FC-BJ:
– max-check is integrated in fc-bj-label and fc-bj-unlabel– Enjoys advantages of FC and BJ… but suffers
malady of BJ (jump the step)
• FC-CBJ: – Best algorithm for far (assuming static variable
ordering)– fc-cbj-label and fc-cbj-unlabel
September 11, 2005 Backtracking 44
Foundations of Constraint Processing, Fall 2005
Consistency checking: summary
• Chronological backtracking– Uses back-checking– No extra data structures
• Backmarking– Uses mcl and mbl– Two types of consistency-checking savings
• Forward-checking– Works more at every instantiation, but expands fewer
subtrees– Uses: reductions[i], future-fc[i], past-fc[i]
September 11, 2005 Backtracking 45
Foundations of Constraint Processing, Fall 2005
Experiments• Empirical evaluations on Zebra
– Representative of design/scheduling problems– 25 variables, 122 binary constraints– Permutation of variable ordering yields new search
spaces– Variable ordering: different bandwidth/induced width
of graph
• 450 problem instances were generated• Each algorithm was applied to each instance
Experiments were carried out under static variable ordering
September 11, 2005 Backtracking 46
Foundations of Constraint Processing, Fall 2005
Analysis of experiments
Algorithms compared with respect to:
1. Number of consistency checks (average)FC-CBJ < FC-BJ < BM-CBJ < FC < CBJ < BMJ < BM < BJ < BT
2. Number of nodes visited (average)FC-CBJ < FC-BJ < FC < BM-CBJ < BMJ =BJ < BM = BT
3. CPU time (average)FC-CBJ < FC-BJ < FC < BM-CBJ < CBJ < BMJ < BJ < BT < BM
FC-CBJ apparently the champion
September 11, 2005 Backtracking 47
Foundations of Constraint Processing, Fall 2005
Additional developments• Other backtracking algorithms exist:
– Graph-based backjumping (GBJ), etc. [Dechter]
– Pseudo-trees [Freuder 85]
• Other look-ahead techniques exist:– DAC, MAC, etc.
• More empirical evaluations: – over randomly generated problems
• Theoretical evaluations: – Based on approach of Kondrak & Van Beek IJCAI’95
September 11, 2005 Backtracking 48
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search • Hybrid backtracking algorithms• Looking ahead
– Forward checking (FC)– Directional Arc Consistency (DAC)– Maintaining Arc Consistency (a.k.a. full arc-
consistency)
• Evaluation of (deterministic) BT search algorithms
September 11, 2005 Backtracking 49
Foundations of Constraint Processing, Fall 2005
Looking ahead• Rationale:
– As decisions are made (conditioning), • Revise the domain of future variables to propagate the
effects of decisions • I.e., eliminate inconsistent choices in future sub-problem
– Domain annihilation of a future variable avoids expansion of useless portions of the tree
• Techniques– Partial: forward-checking (FC), directional arc-
consistency (DAC)– Full: Maintaining arc-consistency (MAC)– Use: Revise(Vf, Vc), Vf future variable, Vc current
variable
September 11, 2005 Backtracking 50
Foundations of Constraint Processing, Fall 2005
Revising the domain of Vi
• Revising the domain of Vi given a constraint C on Vi (i.e., Vi Scope(C))
• General notation: Revise(Vi,C)
• In a binary CSP:
Revise(Vi,C)=
Revise(Vi,CVi,Vj)=
Revise(Vi, Vj)
September 11, 2005 Backtracking 51
Foundations of Constraint Processing, Fall 2005
Revise(Vi, Vj)
• Revise(Vi, Vj)
1. x DVi
2. y DVj
3. If y DVj with (x,y) C return()
4. ElseIf DVi DVi \ {x}
NOTE: only DVi may be updated
September 11, 2005 Backtracking 52
Foundations of Constraint Processing, Fall 2005
Domain filtering in lookahead
• Vc current variable
• Vf future variable
• {Vf} all future variables
• Revise(Vf, Vc)
• FC(Vc):
1. Vf {Vf} connected to Vc
2. Revise(Vf,Vc)
3. If DVf ={} then return(failure)
September 11, 2005 Backtracking 53
Foundations of Constraint Processing, Fall 2005
• FC:
• DAC:– assumes a fixed variable ordering d
• MAC:
– does more pruning at the cost of more consistency checks
Look-ahead techniques: FC, DAC, MAC
FC(Vc); Vf {Vf}, FC(Vf) in the ordering d
FC(Vc); AC({Vf})
FC(Vc)
FC(Vc);
Repeat until quiescence or failure
Vf1,Vf2 {Vf}, Revise(Vf1,Vf2)
September 11, 2005 Backtracking 54
Foundations of Constraint Processing, Fall 2005
Terminology overload alert: FC
• FC is used to denote any of the following:– a partial look-ahead schema – a specific chronological backtrack search
algorithm that uses the partial look-ahead schema
• Meaning is inferred from context• Not a healthy situation, but a reality• Advice: state upfront the meaning of your terms
and stick to them throughout your paper
September 11, 2005 Backtracking 55
Foundations of Constraint Processing, Fall 2005
Outline
• Review of terminology of search • Hybrid backtracking algorithms• Looking ahead• Evaluation of (deterministic) BT search
algorithms [Dechter, 6.6.2]
– CSP parameters– Comparison criteria – Theoretical evaluations– Empirical evaluations
September 11, 2005 Backtracking 56
Foundations of Constraint Processing, Fall 2005
CSP parameters
• Number of variables: n
• Domain size: a, d
• Constraint tightness:
t = |forbidden tuples| / | all tuples |
• Proportion of constraints (a.k.a., constraint density, constraint probability):
p1 = e / emax, e is nbr of constraints
September 11, 2005 Backtracking 57
Foundations of Constraint Processing, Fall 2005
Comparison criteria1. Number of nodes visited (NV)
• Every time you call label
2. Number of constraint check (CC)• Every time you call check(i,j)
3. CPU time• Be as honest and consistent as possible
4. Some specific criterion for assessing the quality of the improvement proposed
Presentation of values:• Average or median of criterion• (qualified) run-time distribution• Solution-quality distribution
September 11, 2005 Backtracking 58
Foundations of Constraint Processing, Fall 2005
Theoretical evaluations
• Comparing NV and/or CC
• Common assumptions: – for finding all solutions
– static orderings
September 11, 2005 Backtracking 59
Foundations of Constraint Processing, Fall 2005
Empirical evaluation: data sets
• Use real-data sets (anecdotal evidence)
• Use benchmarks (csp library)
• Use randomly generated problems
September 11, 2005 Backtracking 60
Foundations of Constraint Processing, Fall 2005
Empirical evaluations: random problems
• Various models exist (use Model B)– Models A, B, C, E, F, etc.
• Vary parameters: <n, a, t, p>– Number of variables: n– Domain size: a, d– Constraint tightness: t = |forbidden tuples| / | all tuples |
– Proportion of constraints (a.k.a., constraint density, constraint probability): p1 = e / emax
• Issues: – Uniformity– Difficulty (phase transition)– Solvability of instances (for incomplete search techniques)
September 11, 2005 Backtracking 61
Foundations of Constraint Processing, Fall 2005
Model B1. Input: n, a, t, p12. Generate n nodes3. Generate a list of n2 tuples of all combinations of 2
nodes4. Choose e elements from above list as constraints to
between the n nodes5. If the graph is not connected, throw away, go back to
step 4, else proceed6. Generate a list of a2 tuples of all combinations of 2
values7. For each constraint, choose randomly a number of
tuples from the list to guarantee tightness t for the constraint
September 11, 2005 Backtracking 62
Foundations of Constraint Processing, Fall 2005
Example: MAC vs. FC
• Reference: [Sabin & Freuder, ECAI94], [Bessière & Régin, CP97], [Sabin & Freuder, CP97], [Gent & Prosser, APES-20-2000], [Experiments by Lin XU, 2001], [Yang, MS thesis 2003]
• Results: (sketchy)Low tightness High tightness
Low density (sparse) FC MAC
High density (dense) FC FC
Note: results depend on
• Variable ordering (static vs. dynamic)
• Problem difficulty (positive relative to crossover point)