J.Stat.M

ech.(2005)

P11016

ournal of Statistical Mechanics:An IOP and SISSA journalJ Theory and Experiment

Survey-propagation decimation throughdistributed local computations

Joel Chavas1, Cyril Furtlehner2, Marc Mezard2 andRiccardo Zecchina3

1 ISI, Viale Settimio Severo 65, 10133 Torino, Italy2 LPTMS, Batiment 100, Universite Paris-Sud, Centre Scientifique d’Orsay,15 rue Georges Clemenceau, 91405 Orsay Cedex, France3 ICTP, Strada Costiera 11, I-34100 Trieste, ItalyE-mail: chavas@isiosf.isi.it, furtlehn@lptms.u-psud.fr, mezard@lptms.u-psud.frand zecchina@ictp.trieste.it

Received 6 June 2005Accepted 6 October 2005Published 30 November 2005

Online at stacks.iop.org/JSTAT/2005/P11016doi:10.1088/1742-5468/2005/11/P11016

Abstract. We discuss the implementation of two distributed solvers of therandom K-SAT problem, based on some development of the recently introducedsurvey propagation (SP) algorithm. The first solver, called the ‘SP diffusionalgorithm’, diffuses as dynamical information the maximum bias over the system,so that variable nodes can decide to freeze in a self-organized way, each variablemaking its decision on the basis of purely local information. The second solver,called the ‘SP reinforcement algorithm’, makes use of time-dependent externalforcing messages on each variable, which are adapted in time in such a way thatthe algorithm approaches its estimated closest solution. Both methods allowus to find a solution of the random 3-SAT problem in a range of parameterscomparable with the best previously described serialized solvers. The simulatedtime of convergence towards a solution (if these solvers were implemented on afully parallel device) grows as log(N).

Keywords: heuristics, message-passing algorithms

c©2005 IOP Publishing Ltd 1742-5468/05/P11016+25$30.00

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Contents

1. Introduction 2

2. The SAT problem and its factor graph representation 4

3. Survey propagation 63.1. Warnings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63.2. The algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4. Distributed SP I: diffusion algorithm 84.1. Simulating SP in parallel . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84.2. Propagation of information . . . . . . . . . . . . . . . . . . . . . . . . . . . 104.3. Experimental results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

4.3.1. Orders of magnitude for the tuning parameters. . . . . . . . . . . . 134.3.2. Dependence on the size of the system. . . . . . . . . . . . . . . . . 144.3.3. Dependence on α. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14

5. Distributed SP II: ‘reinforcement algorithm’ 155.1. SP with external forcing field . . . . . . . . . . . . . . . . . . . . . . . . . 155.2. Presentation of the method . . . . . . . . . . . . . . . . . . . . . . . . . . 175.3. Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

5.3.1. Determination of the optimal intensity of the forcing π. . . . . . . . 195.3.2. Range of validity of the method. . . . . . . . . . . . . . . . . . . . . 205.3.3. Which solutions are addressed? . . . . . . . . . . . . . . . . . . . . 215.3.4. Logarithmic dependence of time on the size of the graph. . . . . . . 225.3.5. Performance of the reinforcement algorithm on a serialized

implementation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

6. Conclusion 23

Acknowledgments 24

References 24

1. Introduction

Recent progress in the statistical physics of disordered systems has provided results ofalgorithmic interest: the connection between the geometrical structure of the space ofsolutions in random NP-complete constraint satisfaction problems (CSPs) and the onset ofexponential regimes in search algorithms [3, 16] has been at least partially clarified. Moreimportant, the analytical methods have been converted into a new class of algorithms [17]with very good performance [11].

Similarly to statistical physics models, a generic CSP is composed of many discretevariables which interact through constraints, where each constraint involves a smallnumber of variables. When CSPs are extracted at random from non-trivial ensemblesthere appear phase transitions as the constraint density (the ratio of constraints to

doi:10.1088/1742-5468/2005/11/P11016 2

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

variables) increases: for small densities the problems are satisfiable with high probability,whereas when the density is larger than a critical value the problems become unsatisfiablewith high probability. Close to the phase boundary on the satisfiable side, most ofthe algorithms are known to take (typically) exponential time to find solutions. Thephysical interpretation for the onset of such exponential regimes in random hard CSPconsists in a trapping of local search processes in metastable states. Depending on themodels and on the details of the process the long time behaviour may be dominated bydifferent types of metastable states. However, a common feature which can be observednumerically is that for simulation times which are sub-exponential in the size of theproblem there exists an extensive gap in the number of violated constraints which separatesthe blocking configurations from the optimal ones. Such behaviour can be tested onconcrete random instances which therefore constitute a computational benchmark formore general algorithms.

In the last few years there has been a great progress in the study of phase transitions inrandom CSP which has produced new algorithmic tools: problems which were consideredto be algorithmically hard for local search algorithms, for instance random K-SAT close toa phase boundary, turned out to be efficiently solved by the so called survey propagation(SP) algorithm [17] arising from the replica symmetry broken (RSB) cavity approach toCSP. According to the statistical physics analysis, close to the phase transition pointthe solution space breaks up into many smaller clusters [16, 17]. Solutions in separateclusters are generally far apart. This picture has been confirmed by rigorous results, firston the simple case of the XORSAT problem [2, 5], and more recently for the satisfiabilityproblem [4, 6]. Moreover, the physics analysis indicates that clusters which correspondto partial solutions—which satisfy some but not all of the constraints—are exponentiallymore numerous than the clusters of complete solutions and act as dynamical traps forlocal search algorithms. SP turns out to be able to deal efficiently with the proliferationof such clusters of metastable states.

The SP algorithm consists in a message-passing technique which is tightly related tothe better known belief propagation algorithm (BP) [21]—recently applied with strik-ing success in the decoding of error-correcting codes based on sparse graph encod-ings [7, 8], [12]–[15]—but which differs from it in some crucial point. The messages sentalong the edges of the graph underlying the combinatorial problem describe in a prob-abilistic way the cluster-to-cluster fluctuations of the optimal assignment for a variable;while BP performs a ‘white’ average over all the solutions, SP makes averages assuminga decomposition of the global probability measure into intra-cluster measures for eachvariable. The order parameter computed by SP is a full list of probability distributionstelling us what is the probability of picking up a cluster at random and finding a givenvariable forced to take a specific value within it (‘frozen’ variable). While BP neglects cor-relations, SP takes care of those correlations that are compatible with the decomposition.Once the iterative equations have reached a fixed-point of such probability distributions(called ‘surveys’ because they capture somehow the distribution of the expectation in thedifferent clusters), it becomes possible in general to identify the variables which can besafely fixed and to simplify the problem accordingly [11, 17]. This procedure, which isintrinsically serial, is known as the SP-inspired decimation algorithm (SID).

From the experimental point of view, the SID has been efficiently used to solvemany instances in the hard region of different satisfiability problems and of the graph

doi:10.1088/1742-5468/2005/11/P11016 3

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

colouring problem, including instances too large for any earlier method [17]. For example,for random 3-SAT, instances close to the threshold, up to sizes of order 107 variableswere solved and the computational time in this regime was found experimentally to scaleroughly as N(log N)2.

In the present manuscript we show how the SP decimation algorithm can be madefully distributed. This opens the possibility of a parallel implementation which would leadto a further drastic reduction in the computational cost.

The paper is organized as follows: in section 2 we introduce some notation and theK-SAT problem, in section 3 we recall the SP algorithm along with the serial decimationprocedure, and sections 4 and 5 are devoted to explaining our new parallel solvers,respectively the first one based on information diffusion and the second one based onexternal forcing on the variables.

2. The SAT problem and its factor graph representation

Though the results we shall discuss are expected to hold for many types of CSP, here weconsider just one representative case: the K-SAT problem.

A K-SAT formula consists of N Boolean variables xi ∈ {0, 1}, i ∈ {1, . . . , N}, withM constraints, in which each constraint is a clause, which is the logical OR (∨) of thevariables it contains or of their negations.

Solving the K-SAT formula means finding an assignment of the xi ∈ {0, 1} ={directed, negated} which is such that all the M clauses are true. In the literature, suchan assignment can indifferently be called the solution, satisfying assignment or groundstate if one uses the statistical physics jargon.

In the case of randomly generated K-SAT formulae (variables appearing in the clauseschosen uniformly at random without repetitions and negated with probability 1/2), asophisticated phase transition phenomenon sets in: when the number of constraintsM = αN is small, the solutions of the K-SAT formulae are distributed close one to eachother over the whole N -dimensional space, and the problem can easily be solved by theuse of classical local search algorithms. Conversely, when α is included in a narrow regionαd < α < αc, the problem is still satisfiable but the now limited solution phase breaksdown into an exponential number of clustered components. Solutions become groupedtogether into clusters which are far apart one from the other. This range of parametersis known as the hard-SAT phase: classical local search algorithms get trapped by strictlypositive energy configurations which are exponentially more numerous than the groundstate ones.

A clause a is characterized by the set of variables i1, . . . , iK which it contains, andthe list of those which are negated, which can be characterized by a set of K numbersJa

ir ∈ {±1} as follows. The clause is written as

(zi1 ∨ · · · ∨ zir ∨ · · · ∨ ziK ) (1)

doi:10.1088/1742-5468/2005/11/P11016 4

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

V (j)sa

j

ub–>j

hj–>a

ua–>i

k

ab

i

V (j)au

Figure 1. A function node a and its neighbourhood. The survey of the cavity-biasua→i can be computed from the knowledge of the joint probability distributionfor all the cavity-biases in the set U , i.e. those coming onto all variable node jneighbours of a (except j = i).

where zir = xir if Jair = −1 and zir = xir if Ja

ir = 1 (note that a positive literal isrepresented by a negative J). The problem is to find whether there exists an assignmentof the xi ∈ {0, 1} which is such that all the M clauses are true. We define the total costC of a configuration x = (x1, . . . , xN) as the number of violated clauses.

In the so called ‘factor graph’ representation, the SAT problem can be depicted asfollows (see figure 1). Each of the N variables is associated to a vertex in the graph, calleda ‘variable node’ (circles in the graphical representation), and each of the M clauses isassociated to another type of vertex in the graph, called a ‘function node’ (squares inthe graphical representation). A function node a is connected to a variable node i by anedge whenever the variable xi (or its negation) appears in the clause a. In the graphicalrepresentation, we use a full line between a and i whenever the variable appearing in theclause is xi (i.e. Ja

i = −1) and a dashed line whenever the variable appearing in the clauseis xi (i.e. Ja

i = 1).

Throughout this paper, the variable node indices are taken in i, j, k, . . ., while thefunction node indices are taken in a, b, c, . . .. For every variable node i, we denote by V (i)the set of function nodes a to which it is connected by an edge, by ni = |V (i)| the degreeof the node, by V+(i) the subset of V (i) consisting of function nodes a where the variableappears un-negated—the edge (a, i) is a full line, and by V−(i), the complementary subsetof V (i), consisting of function nodes a where the variable appears negated—the edge(a, i) is a dashed line. V (i) \ b denotes the set V (i) without a node b. Similarly, for eachfunction node a, we denote by V (a) = V+(a) ∪ V−(a) the set of neighbouring variable

doi:10.1088/1742-5468/2005/11/P11016 5

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

nodes, decomposed according to the type of edge connecting a and i, and by na its degree.Given a function node a and a variable node j, connected by an edge, it is also convenientto define the two sets V s

a (j) and V ua (j), where the indices s and u respectively refer to the

neighbours which tend to make variable j respectively satisfy or not satisfy the clause a,defined as (see figure 1)

if Jaj = 1 : V u

a (j) = V+(j); V sa (j) = V−(j) \ a (2)

if Jaj = −1 : V u

a (j) = V−(j); V sa (j) = V+(j) \ a. (3)

3. Survey propagation

The survey propagation algorithm (SP) is a message passing algorithm in whichthe messages passed between function nodes and variable nodes have a probabilisticinterpretation. Every message is the probability distribution of some warnings. In thissection we provide a short self-contained introduction to SP, starting with the definition ofwarnings. We refer to [11, 16, 17] for the original derivation of SP from statistical physicstools.

3.1. Warnings

The basic elementary message passed from one function node a to a variable i (connectedby an edge) is a Boolean number ua→i ∈ {0, 1} called a ‘warning’.

Given a function node a and one of its variable nodes i, the warning ua→i is determinedfrom the warnings ub→j arriving on all the variables j ∈ V (a) \ i according to

ua→i =∏

j∈V (a)\i

θ

−Jaj

∑

b∈V (j)\a

J bj ub→j

, (4)

where θ(x) = 0 if x ≤ 0 and θ(x) = 1 if x > 0.A warning ua→i = 1 can be interpreted as a message sent from function node a, telling

the variable i that it should adopt the correct value in order to satisfy clause a. This isdecided by a according to the warnings which it has received from all the other variablesj to which it is connected: if (

∑b∈V (j)\a J b

j ub→j)Jaj < 0, this means that the tendency for

site j (in the absence of a) would be to take a value which does not satisfy clause a. If allneighbours j ∈ V (a) \ i are in this situation, then a sends a warning to i.

The simplest message passing algorithm one can think of is just the propagation ofwarnings, in which the rule (4) is used as an update rule which is implemented sequentially.The interest in WP largely comes from the fact that it gives the exact solution for tree-problems. In more complicated problems where the factor graph is not a tree, it turnsout that WP is unable to converge whenever the density of constraints is so large that thesolution space is clustered. This has prompted the development of the more elaborate SPmessage passing procedure, which also appears to give good results for locally tree-likegraphs with loops.

doi:10.1088/1742-5468/2005/11/P11016 6

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

3.2. The algorithm

A message of SP, called a survey, passed from one function node a to a variable i (connectedby an edge), is a real number ηa→i ∈ [0, 1]. The interpretation of the survey ηa→i is aprobability among all clusters of satisfying assignments that a warning is sent from ato i.

The SP algorithm uses a random sequential update (see algorithm 1), which calls thelocal update rule SP-UPDATE (algorithm 2):

Algorithm 1 : SP

Input: The factor graph of a Boolean formula in conjunctive normal form; amaximal number of iterations tmax; a requested precision εOutput: UN-CONVERGED if SP has not converged after tmax sweeps. If it hasconverged: the set of all messages η∗a→i

1: At time t = 0:2: for every edge a → i of the factor graph do3: randomly initialize the messages ηa→i(t = 0) ∈ [0, 1]4: end for5: for t = 1 to t = tmax do6: sweep the set of edges in a random order, and update sequentially the

warnings on all the edges of the graph, generating the values ηa→i(t),using subroutine SP-UPDATE

7: if |ηa→i(t) − ηa→i(t − 1)| < ε on all the edges then8: the iteration has converged and generated η∗a→i = ηa→i(t)9: goto line 1210: end if11: end for12: if t = tmax then13: return UN-CONVERGED14: else {t < tmax}15: return the set of fixed point warnings η∗a→i = ηa→i(t)16: end if

Whenever the SP algorithm converges to a fixed-point set of messages η∗a→i, one

can use it in a decimation procedure in order to find a satisfiable assignment, if suchan assignment exists. This procedure, called the survey inspired decimation (SID), isdescribed in algorithm 3.

There exist several variants of this algorithm. In the code which is available at [23],for performance reasons we update simultaneously all η belonging to the same clause. Theclauses to be updated are chosen in a random permutation order at each iteration step.The algorithm can also be randomized by fixing, instead of the most biased variables, onevariable randomly chosen in the set of the x per cent variables with the largest bias. Thisstrategy allows us to use some restart in the case where the algorithm has not found asolution. A fastest decimation can also be obtained by fixing in step 12, instead of onevariable, a fraction f of the Nt variables (the most polarized ones) which have not yetbeen fixed (going back to one variable when fNt < 1).

doi:10.1088/1742-5468/2005/11/P11016 7

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Algorithm 2 : subroutine SP-UPDATE(ηa→i)

Input: Set of all messages arriving onto each variable node j ∈ V (a) \ iOutput: New value for the message ηa→i

1: for every j ∈ V (a) \ i do2: compute the three numbers:

Πuj→a =

1 −∏

b∈V ua (j)

(1 − ηb→j)

∏

b∈V sa (j)

(1 − ηb→j)

Πsj→a =

1 −∏

b∈V sa (j)

(1 − ηb→j)

∏

b∈V ua (j)

(1 − ηb→j)

Π0j→a =

∏

b∈V (j)\a(1 − ηb→j) (5)

if a set like V sa (j) is empty, the corresponding product takes value 1 by

definition.3: Using these numbers , compute and return the new survey:

ηa→i =∏

j∈V (a)\i

[Πu

j→a

Πuj→a + Πs

j→a + Π0j→a

]. (6)

If V (a) \ i is empty, then ηa→i = 1.4: end for

In [11], experimental results are reported which show that SID is able to solveefficiently huge SAT instances very close to the threshold. More specifically, the datadiscussed in [11] show that if SID fixes at each step only one variable it converges inΘ(N2 log N) operations (the time taken by walksat [18] to solve the simplified sub-formulaseems to grow more slowly). When a fraction of variables is fixed at each time step, weget a further reduction of the cost to O(N(log N)2) (the second log comes from sortingthe biases).

In the following sections we show how the SP procedures can be made fully distributedand hence amenable to a parallel implementation.

4. Distributed SP I: diffusion algorithm

4.1. Simulating SP in parallel

The SP algorithm described in the preceding section is by construction a distributedalgorithms, since updates of nodes are performed using message passing procedures fromnearest neighbours. However, when one uses the surveys in order to find the SATassignments, in the SID procedure, the decimation process breaks this local informationexchange design: it requires global information, namely the maximal polarization field,used to decide which node has to be frozen at first. It is the purpose of this section todefine a procedure able to diffuse such a global information, by using the message passingprocedure.

doi:10.1088/1742-5468/2005/11/P11016 8

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Algorithm 3 : SID

Input: The factor graph of a Boolean formula in conjunctive normal form. A maximal number of iterations

tmax and a precision ε used in SP

Output: One assignment which satisfies all clauses, or ’SP UNCONVERGED’, or ’probably UNSAT’

1: Random initial condition for the surveys

2: run SP

3: if SP does not converge then4: return ’SP UNCONVERGED’ and stop (or restart, i.e.goto line 1).

5: else {if SP converges}6: keep the fixed-point surveys η∗

a→i; goto line 9

7: end if8: Decimate:

9: if non-trivial surveys ({η �= 0}) are found then10: for each variable node i do11: evaluate the three ’biases’ {W (+)

i , W(−)i , W

(0)i } defined by:

W(+)i =

Π+i

Π+i + Π−

i + Π0i

(7)

W(−)i =

Π−i

Π+i + Π−

i + Π0i

(8)

W(0)i = 1 − W

(+)i − W

(−)i (9)

where Π+i , Π−

i , Π0i are defined by

Π+i =

1 −∏

a∈V+(i)

(1 − η∗a→i)

∏

a∈V−(i)

(1 − η∗a→i)

Π−i =

1 −∏

a∈V−(i)

(1 − η∗a→i)

∏

a∈V+(i)

(1 − η∗a→i)

Π0i =

∏

a∈V (i)

(1 − η∗a→i) (10)

12: fix the variable with the largest |W (+)i − W

(−)i | to the value xi = 1 if W

(+)i > W

(−)i , to the value

xi = 0 if W(+)i < W

(−)i

13: Clean the graph, which means: to remove the clauses satisfied by this fixing, to reduce the

clauses that involve the fixed variable with opposite literal, and to update the number of unfixed

variables.

14: end for15: else {if all surveys are trivial ({η = 0}) }16: output the simplified sub-formula and run on it a local search process (e.g. walksat [18]).

17: end if18: if the problem is solved completely by unit clause propagation then19: return ‘‘SAT’’ and stop20: else if no contradiction is found then21: continue the decimation process on the smaller problem (goto line 2)

22: else {if a contradiction is reached}23: return ’probably UNSAT’

24: stop25: end if

In the present distributed implementation of SP (see algorithm 4), each node isresponsible for its own information; any sort of centralized information is prohibited. Theinformation stored at a given function node a is the set of surveys {ηa→i, i ∈ V (a)}.In turn, a given variable node i keeps information which is the set of cavity-fields{Πi→a, a ∈ V (i)}. The scheme amounts to implementing a message passing procedure suchthat function nodes send their {η} values to neighbouring variable nodes, and variablenodes send their field values in return. Each time a node receives a message from its

doi:10.1088/1742-5468/2005/11/P11016 9

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

neighbours, it has to update its own information, according to the survey propagationequations. The broadcasting of information for each individual node is supposed tooccur at random. Our simulation proceeds by random updating of each variable orfunction node. The update timescale, that is the rate at which nodes in a distributedimplementation are able to collect new information and perform their own update, willbe denoted by τ . Typically, as we shall see, the full decimation process is of the order of104τ , which means that on average each node has to update 104 times before the processis completed.

Algorithm 4 : diffusive decimation algorithm

Input: The factor graph of a Boolean formula in conjunctive normal form; amaximal computational time tmax, a requested precision ε, a number nsu ofsuccessive update with maximum polarization required to freeze a variable, thedamping factor δ for information decay, a random set of cavity biases ηa→i.Output: UN-CONVERGED if the algorithm has not converged after tmax. If it hasconverged: one assignment which satisfy a fraction of the clauses, and aremaining reduced factor graph corresponding to a paramagnetic state which canbe solved calling walksat

1: At time t = 0:2: Initialize the counter nsu(i) = 0, i = 1 . . . N3: parallel update of each variable node using VARIABLE-NODE-UPDATE.4: for t < tmax AND Hmax > ε do5: parallel asynchronized update of each variable node using

VARIABLE-NODE-UPDATE and each function node using FUNCTION-NODE-UPDATE,at a mean rate of time τ per node.

6: end for7: if Hmax < ε then8: return the set of assignments of the frozen variables and the remaining

reduced factor graph of unfrozen variables.9: else10: return UN-CONVERGED11: end if

4.2. Propagation of information

Messages can contain additional information to the one needed to run sp. In thepresent case, the decimation procedure diffuses the information concerning the highestpolarization fields Hmax = max(|W+ − W−|) in the system, in order that the variablenode, being aware of being the most polarized one, can decide to freeze its assignment tothe one given by the orientation of the polarization field.

Static information can freely propagate by simply diffusing in the network, and anycomponent of the network should be informed by a time which has to be proportionalto log(N), N being the size (number of variables nodes) of the system. The difficultyis that the information to be broadcast may still vary with time before the system hasreached equilibrium. The difficulty in propagating a signal saying that the system has

doi:10.1088/1742-5468/2005/11/P11016 10

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Algorithm 5 : subroutine VARIABLE-NODE-UPDATE

Input: Set of all cavity bias surveys {ηa→i, a ∈ V (i)} arriving at a givenvariable node i, information Hmax(a) on the maximum polarization coming fromeach neighbor a ∈ V (i)Output: New value for the set of probabilities {Π±

i→a,Π0i→a, a ∈ V (i)}, new

information Hmax(i) on the maximum polarization as estimated by i

1: for every a ∈ V (i) do2: compute the values of Π±

i→a,Π0i→a using eq. (5).

3: end for4: evaluate the two ’biases’ {W (+)

i ,W(−)i } using eq. (7,8):

5: if Hmax(a) < |W (+)i − W

(−)i | + ε then

6: Hmax(i) = |W (+)i − W

(−)i |

7: nsu(i) = nsu(i) + 18: if nsu(i) > nsu then

9: freeze variable i to the value xi = 1 if W(+)i > W

(−)i , to the value xi = 0

if W(+)i < W

(−)i

10: end if11: else12: Hmax(i) = Max(Hmax(i),Hmax(a)) ∗ (1 − δ)13: nsu(i) = 014: end if

Algorithm 6 : subroutine FUNCTION-NODE-UPDATE

Input: Set of all probabilities {Π±i→a,Π

0i→a, i ∈ V (a)} arriving at a given

function node a, information Hmax(i) on the maximum polarization coming fromeach neighbor i ∈ V (a)Output: New value for the set of cavity bias survey {ηa→i, i ∈ V (a)}, newinformation Hmax(a) on the maximum polarization as estimated by a

1: for every i ∈ V (a) do2: Compute ηa→i using eq. (6).3: Hmax(a) = Max(Hmax(i),Hmax(a)) ∗ (1 − δ)4: end for

converged might be even more severe. Indeed, this amounts to propagating the maximumdisplacement from the last update of each individual node, which is in essence transientinformation.

To take advantage of the basic message-passing procedure, in the case in which theinformation to be diffused can vary with time, we have to add a mechanism able tosuppress obsolete information. The way we propose to do this is to incorporate somedamping factor in the broadcast values.

Let us call δ � 1 this damping factor. The procedure goes as follows (see algorithm 5).

doi:10.1088/1742-5468/2005/11/P11016 11

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Each node (a for a function node or i for a variable node) stores its own estimate of themaximum polarization field Hmax in a local variable Hmax(i) or Hmax(a). Consider firsta variable node i. When this updates, in addition to the set {ηb→i} from its neighbourfunction nodes b, it also collects the information Hmax(b) about the maximum, comingfrom these nodes, and compares it to its own estimation Hmax(i) and also to its own fieldvalue Wi = |W+

i − W−i |. If the new Hmax(i) estimation indicates that i is not the most

polarized variable node (Hmax(i) > Wi), Hmax(i) is multiplied by a factor (1 − δ). Ifinstead it appears that i might be the most polarized node (Hmax(i) = Wi), then Hmax(i)is kept as it stands, with no damping correction. For a function node, when it updates,we apply this damping factor anyway. The virtue of this damping effect is to eliminateany false information. Suppose indeed that at a given time a strongly polarized variablenode diffuses in the network a very high value of Hmax; if after some time this value is notconfirmed (because this node converges to a less polarized state), which means that alllocal fields Wi will be smaller than Hmax, and no unit in the network can pretend to bethe most polarized one. Then mechanically the estimation on the maximum will decay ata rate δ, until a variable node presents a polarization field higher than the decaying falseinformation.

The counterpart is that the damping takes effect spatially on the network. Sotypically, since the radius of the network scales like log(N), the distribution of theestimated maximum given by the set {Hmax(i), Hmax(a)} taken over the network willbe enlarged by some factor δ log(N).

As a result, when playing with the parameter δ, two contradictory effects are at work:

• either the information diffuses rapidly with a poor precision (large δ),

• or it takes a long time before precise information is diffused (small δ).

When it comes to the decimation procedure, this can be actually turned into an advantage,since it provides us directly with a parameter able to fix, in a heuristic manner, the rateat which the decimation is performed. Each variable node is equipped with a countingvariable nsu(i) initially set to zero and counting successive updates with the following twoconditions:

• the variation since last update of the Πi’s is less than ε (convergence criteria, typicallyε � 10−3),

• i may consider itself to be the most polarized variable (Hmax(i) = |W+i − W−

i |).

When one of these conditions is not satisfied, nsu(i) is set to zero. If nsu(i) is greater thana parameter nsu(i), the variable is allowed to freeze in the direction given by W+

i − W−i .

4.3. Experimental results

We have simulated the behaviour of the algorithm using a standard computer, whichallows us to monitor how much progress in computing time would be brought by its fullyparallel implementation, and to optimize the choices of parameters.

We use the random 3-sat formulae as a benchmark problem. Let N denote thenumber of variable nodes and α = M/N the ratio of the number of function nodes Mover the number of variable nodes. The range of interest is α ∈ [4, 4.25]. Simulationsare performed with samples of 105 function nodes. Figures 2–5 present some average over

doi:10.1088/1742-5468/2005/11/P11016 12

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

0 100 200 300 400time/100tau

0

0,5

1ac

tive

clau

sedelta=0.01delta = 0.008delta = 0.006delta = 0.004delta = 0.002delta=0.001

Figure 2. Number of active clauses as a function of time for different values of δfor a 3-sat problem of size N = 105 at α = 4.2 and nsu = 20.

numerous decimation runs, with various parameters. The runs stop when a paramagneticstate is reached, or when the entropy becomes negative. In order to assess the performanceof the algorithm, we measure the simulated time (the time it would take if the computationwas really distributed) to reach a paramagnetic state, which can be handled afterwardsby walksat in order to find a solution.

The observables which we consider are the following.

• Na, the number of active (unfrozen) variable nodes.

• Ma, the number of active function nodes.

• α2, the ratio Ma2/Na, where Ma2 is the number of active two-clauses.

• α3, the ratio Ma3/Na, where Ma3 is the number of active three-clauses. Since we startfrom a 3-SAT problem we have of course Ma = Ma2 + Ma3.

• The entropy of the system, computed from the expression given in [11].

4.3.1. Orders of magnitude for the tuning parameters. The algorithm depends on twotunable parameters.

• The damping factor δ on the diffusion of information. We find that δ ∈ [0.001, 0.01] isroughly the range of effectiveness of this parameter for N = 105. This actually givesessentially the decimation rate between two convergences of sp. For δ ≥ 0.01, toomany variables (�103) are frozen at the same time, and the algorithm ceases to finda paramagnetic state. Instead, when δ < 0.001, variables are decimated one by one.The time dependence of the decimation process with this parameter is illustrated infigure 2.

doi:10.1088/1742-5468/2005/11/P11016 13

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

1000 10000 1e+05N

30

40

50

60

70

80

90fi

nal

time/

100t

au

Figure 3. Decimation time up to the paramagnetic state as a function of the sizeof the system, for a 3-sat problem at α = 4.2, δ = 0.005 and nsu = 20.

• nsu, the number of successive successful updates needed for a variable to get frozen. Itdepends on the precision ε which is required. For ε = 10−3 it is found that the rangeof validity of this parameter is between 10 and 50. Above 50, sp converges before thenext decimation cascade. Below 20 the decimation process is essentially continuous.Below 10 the algorithm ceases to converge to a paramagnetic state.

A reasonable choice for the selection of parameters leading to a fast but still successfuldecimation procedure is around δ = 0.01 and nsu = 10.

4.3.2. Dependence on the size of the system. Once the typical values for the tunableparameters are determined, it is possible to study the behaviour of the algorithm withthe size of the problem, N . Figure 3 shows that the dependence on N is of the order oflog N , which is the best performance one could have with such an algorithm. Anotherexpected result is that rate of successful decimation should increase when N increases,since finite size effects are known to alter the efficiency of algorithms for solving 3-sat

formulae. This is what is effectively observed in figure 4.

4.3.3. Dependence on α. Depending on α we observe that when we approach thetransition point αc � 4.26 the final decimation rate increases (see figure 5), which isconsistent with the hypothesis that the size of the backbone of solution clusters increaseswhen we approach the critical value of α. This can also be visualized when looking atthe phase space trajectory of the decimation process (see figure 6). During the processtwo-clauses are generated, which leads to representation of the trajectory in the plane(α2, α3). The trajectory is shorter when we go away from αc.

doi:10.1088/1742-5468/2005/11/P11016 14

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

1000 10000 1e+05N

0

0,5

1so

lvin

g ra

te

Figure 4. Rate of successful decimation process as a function of N for a 3-satproblem at α = 4.2, δ = 0.005 and nsu = 20.

5. Distributed SP II: ‘reinforcement algorithm’

5.1. SP with external forcing field

The SP procedure has recently been generalized to allow the retrieval of a large number ofdifferent solutions thanks to the introduction of probability preconditionings [9, 10]. Theoriginal factor graph is modified applying to each variable i a constant external forcingsurvey in direction ξi ∈ {−1, 1} and of intensity ηext

i = π. An a priori probability π ofassuming the value ±1 is thus being assigned to the variable i. The components of theforcing modify the usual SP equation (5) in the following way:

Π±j→a =

1 −(1 − πδξj ,±

) ∏

b∈V±(j)\a

(1 − ηb→j)

(1 − πδξj ,∓

) ∏

b∈V∓(j)

(1 − ηb→j)

Π0j→a =

(1 − πδξj ,+

) (1 − πδξj ,−

) ∏

b∈V (j)\a

(1 − ηb→j) .

(11)

These relations, together with (6), form the SP equations with external messages

(SP-ext). The quantities Π+i , Π−

i , Π0i used to define the local biases on each variable i (see

equation (10)) are modified in a similar manner and expressed as a function of the new

doi:10.1088/1742-5468/2005/11/P11016 15

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

0 10 20time/100tau

0

0,5

1A

ctiv

e cl

ause

s4

30

alpha = alpha = 4.05alpha=4.1alpha=4.15alpha=4.2alpha=4.25

Figure 5. Number of active clauses as a function of time during the decimationprocess for different values of α, for a 3-sat problem of size N = 105 at δ = 0.01and nsu = 10.

0 0,5 1 1,5

alpha2

0

1

2

3

4

5

alph

a3

4.254.24.154.14.054

Figure 6. Trajectories in the phase space (α2, α3) during the decimation processfor various values of α for a 3-sat problem of size N = 105 at δ = 0.01 andnsu = 10. (Five samples for each value of α; all decimation processes weresuccessful up to α = 4.2, and all failed for α = 4.25.) The slope of the linear partof the trajectories is � −2.4.

doi:10.1088/1742-5468/2005/11/P11016 16

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

probabilities η∗b→j at the fixed point:

Π±j→a =

1 −(1 − πδξj ,±

) ∏

b∈V±(j)

(1 − η∗

b→j

)

(1 − πδξj ,∓

) ∏

b∈V∓(j)

(1 − η∗

b→j

)

Π0j→a =

(1 − πδξj ,+

) (1 − πδξj ,−

) ∏

b∈V (j)

(1 − η∗

b→j

).

(12)

The external messages drive the value of the surveys η∗a→i towards the clusters

maximally aligned with the external vector field. π = 0 corresponds to the case inwhich the external messages are absent; in the case π = 1, the system is fully forced in

the direction ξ. When π is correctly chosen, this enables us to address more specifically(compared to the SID algorithm, which permits us to retrieve only one solution) clustersclose to a given region of the N -dimensional space. Thus, a right choice of the a priori

probability π, and the use of alternative random directions ξ, invariably permit us toretrieve many solutions in many different clusters.

The criterion used to decimate towards a solution is still the global criterion of themaximum of the magnetization over all variables. So, the decimation based on the SP-extequations is not yet a distributed solver of the K-SAT formulae.

5.2. Presentation of the method

In order to make the decimation procedure fully distributed, we may adopt time-dependentexternal messages which are updated according to a reinforcement strategy. Moreprecisely, good properties of convergence are obtained adopting the following scheme:

• in one time step (sweep), all the ηs are updated in parallel using the SP equationswith external messages (6), (11), which in this scheme are time-dependent;

• every second time step, one calculates all temporary local fields {W (+)i , W

(−)i , W

(0)i }

using the equations (7)–(9), (13)—note that these are also functions of the externalmessages. The directions ξi, ∀i ∈ {1, . . . , N} of the temporary local messages, whichare now time dependent, are updated in parallel to equal sgn(W+

i −W−i ), i.e. to align

itself with the local bias W+i − W−

i .

As we shall see, with an appropriate choice of the intensity π of the forcing, mostvariables are completely polarized at the end of a single convergence. A solution of the K-SAT formula is finally found by fixing each variable in the direction of its local bias. Thereinforcement algorithm (RA) defined above is purely local and hence gives an efficientdistributed solver for random K-SAT formulae in the hard-SAT phase.

The algorithm is precisely described in algorithm 7. Its interpretation and apreliminary study of some of its properties are discussed in the subsequent sections.

To understand how the reinforcement strategy is working, one has to look back at theinterpretation of the SP equations with an external forcing survey: imposing an externalforcing survey of direction ξi ∈ {−1, 1} on the variable i tries to drive the solution ofthe cavity bias distributions towards the clusters with a local orientation matching theexternally imposed direction ξi.

doi:10.1088/1742-5468/2005/11/P11016 17

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Algorithm 7 : reinforcement algorithm

Input: The factor graph of a Boolean formula in conjunctive normal form; amaximal number of iterations tmax, a requested precision ε and the probabilityπ that the external message sends a warningOutput: UN-CONVERGED if the algorithm has not converged after tmax sweeps. Ifit has converged: one assignment which satisfies all clauses

1: At time t = 0:2: for every edge a → i of the factor graph do3: randomly initialize the cavity biases ηa→i(t = 0) ∈ [0, 1]. The additional

surveys are initially set to Qi(�ui, 0)4: end for5: for t = 1 to t = tmax do6: update in parallel the cavity bias surveys on all the edges of the graph,

generating the values ηa→i(t), using subroutine CBS-UPDATE.7: half of the times, update in parallel the reference direction ξi ∈ {−1, 1}

for all variables xi, using subroutine FORC-UPDATE.8: if |ηa→i(t) − ηa→i(t − 1)| < ε on all the edges then9: the iteration has converged and generated η∗a→i = ηa→i(t)10: goto line 1311: end if12: end for13: if t = tmax then14: return UN-CONVERGED15: else {If t < tmax}16: return the satisfying assignment which is obtained by fixing the boolean

variable xi parallel to its local field : xi = sign{W (+)i − W

(−)i }, and by

performing the simplifications over the neighboring variables17: end if

Algorithm 8 : subroutine CBS-UPDATE(ηa→i)

Input: Set of all cavity bias surveys arriving onto each variable nodej ∈ V (a) \ iOutput: New value for the cavity bias survey ηa→i

1: for every j ∈ V (a) \ i do2: compute the values of Π±

j→a,Π∓j→a,Π

0j→a using eq. (11).

3: Compute ηa→i using eq. (6).4: end for

Then, choosing ξ parallel to the vector of the temporary local fields is a strategy whichtypically guarantees that, if the intensity π of the forcing is small enough, the overlap

between the forcing vector field ξ and the closest solution is continuously increasing, in astronger way than in the case of the usual SP equations.

doi:10.1088/1742-5468/2005/11/P11016 18

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

Algorithm 9 : subroutine FORC-UPDATE(ξi)

Input: Set of all cavity bias surveys arriving onto the variable node i,including the additional survey Qi(�eξi

;π)Output: New value for the direction ξi of the additional survey

1: Compute the local fields W(+)i ,W

(−)i ,W

(0)i using eq. (7, 8, 9, 12)

2: Compute ξi = sign(W (+)i − W

(−)i )

When π is correctly chosen, some satisfying assignments tend to become stable fixedpoints of the RA dynamics. The optimal value of π is a trade-off between the needfor convergence of the SP equations with external forcing survey, the strength of thestabilization induced on the satisfying assignments by the update rule and the amplitude ofthe perturbation brought by the external forcing. There is a priori no guarantee that thereexists a particular value of the intensity π for which the algorithm, starting from randominitial conditions, gets trapped by a single cluster, and eventually converges to a singlesolution inside this cluster, without any further decimation. But, our experimental resultsshow that, for N big enough and for almost all values of the parameter α correspondingto the SAT phase, there exists an intensity π for which the overlap between the set oflocal fields and the closest satisfying assignment increases until it eventually reaches unityat the end of a unique convergence.

5.3. Results

5.3.1. Determination of the optimal intensity of the forcing π. The efficiency of the methodis tested on random 3-SAT formulae. One first determines the optimal intensity πopt,which is the intensity of the forcing which permits us to find a satisfying assignment inthe minimum number of steps. When the density of ground state clusters increases (forlower values of α), πopt should be set at a higher value to be able to bias the convergencetowards a unique cluster; however, a too high value of π may let the algorithm converge toorapidly in different parts of the graph towards assignments belonging to different clusters.The algorithm would then loop without convergence among these clusters. For these tworeasons, πopt should be positively correlated with the complexity Σ = log(NCl)/N .

One observes that the reinforcement process diverges if the parallel update of theforcing messages is performed at a too high frequency with respect to the frequency of theparallel updates of the η; the local loops in which the algorithm gets trapped are likely dueto simultaneous convergence towards different satisfying assignments in different parts ofthe graph. A good strategy, systematically used for the experiments of the present paper,is to perform the parallel updates of the forcing messages once every second parallel updateof the η.

In the hard-SAT phase, one determines πopt as a function of the calculatedcomplexity [17]. As predicted, the experiments show that, for αD < α < 4.245, πopt

is positively correlated, and on average scales linearly, with Σ (πopt = 11.1×Σ, cf figure 7,each point representing a different formula), and that the probability of converging towardsa satisfying assignment using this value tends to unity (N = 105, ε = 10−3, tmax = 1000;cf figure 8).

doi:10.1088/1742-5468/2005/11/P11016 19

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

0 500 1000 1500

E

t

0

0.05

0.1

0.15

0.2

0.25

0.3

0.35

0.4

0 0.005 0.01 0.015 0.02

Σ, complexity

π = 11.1*Σ

0

200

400

600

Σ=0.0013

π, o

ptim

ized

0.025

Figure 7. Scaling of the optimized π against the complexity Σ. The optimized π,which is the intensity of the forcing permitting us to find a satisfying assignmentin the minimum number of steps, is recalculated for each formula: it dependson average linearly on the complexity: π = 11.1 × Σ (N = 105, K = 3,4.0 < α < 4.24). Each point corresponds to a different formula; note thatthe relationship πopt = f(Σ) becomes slightly sub-linear for Σ < 0.003. Thesub-plot shows the evolution of the energy of the dump solution (in which eachvariable is fixed parallel to the temporary local field) that would be obtained ifthe convergence was stopped at time t (α = 4.24).

5.3.2. Range of validity of the method. In this section we compare RA with the so-calledserialized survey propagation-inspired decimation (SID).

Figure 8 plots the percentage of successful runs (i.e. of convergence towards asatisfying assignment after a single run) of both the synchronous and the asynchronousRAs as a function of α. One calls the algorithm previously described (algorithm 7), inwhich the updates of all the η, as well as respectively the updates of all forcing messages,are performed simultaneously, synchronous RA. Conversely, in the asynchronous RA, theupdates are performed sequentially in a random permutation order (note that, in thelatter case, forcing messages can be updated as frequently as the η). For α = 4.24, bothalgorithms invariably permit us to retrieve a solution. This is in accordance with the resultpresented in [11], in which the classical SID has been shown to always find a solution byfixing 0.125% of variables after each convergence of the SP equations.

The cut-off value of the complexity Σ, for which the decimation converges in halfof the cases, is Σ1/2 = 0.0013 (sub-plot of figure 8). Σ1/2 was constant for all sizes ofthe graph N > 2 × 104. When the complexity decreases, the decimation progressivelyfails. Depending on the value of π, either it converges towards a set of local fields, whichcorresponds to a weighted average of the contributions of numerous clusters of solutions,

doi:10.1088/1742-5468/2005/11/P11016 20

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

0

50

100

4.21 4.22 4.23 4.24 4.25 4.26

%

α

synchronous RAasynchronous RA

0

50

100

0 0.001 0.002 0.003

%

Σ

Figure 8. Percentage of successful runs (convergence leading to a satisfyingassignment) as a function of α. The sub-plot shows the same percentage as afunction of the calculated complexity Σ. The cut-off complexity, for which theRA is successful in half of the cases tested, is Σ1/2 = 0.0013 (N = 105, ε = 10−3,tmax = 1500). The range of validity of the reinforcement algorithm thus matchesthe one of the SID (see text).

or it loops among low energy configurations (usually, for N = 105, α = 4.25, one finallyfinds assignments of energy E = 0–50).

Moreover, asymptotically, the numerical critical α above which the SID, withoutbacktracking, should not permit us to find a satisfying assignment has been estimated tobe αA = 4.252 [19], substantially smaller than the theoretical critical value αC = 4.2667(for the improvement brought by the backtracking procedure, see [11, 20]).

One may study the behaviour of the RA at αA for N = 106. The calculated complexityof the formulae with N = 106 and α = 4.252 equals Σ = 0.001 33±0.000 13, in agreementwith the cut-off complexity determined for formulae of size N = 105 (figure 8). ForN = 106 and α = 4.252, one then sets πopt = 10.5 × Σ (in agreement with the curveof figure 7 for N = 105). The ‘reinforcement algorithm’ using these parameters nicelypermits us to converge towards a solution in 10 out of the 15 formulae tested.

One concludes that the present algorithm is valid at least in the same range ofparameters as the classical SID [11].

5.3.3. Which solutions are addressed? The satisfying assignments found by the presentalgorithm are highly dependent on the initial conditions, i.e. on the initial random valuesimposed on the η. For different random initial conditions, one converges towards solutionsbelonging to different clusters.

To know where the found solutions are located, one chooses an instance from arandom graph ensemble in which the number of clauses where any given variable appears

doi:10.1088/1742-5468/2005/11/P11016 21

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

negated or directed are kept strictly equal (bar-balanced formulae). For such an instance(N = 30 000, α = 3.33), the clusters of solutions are expected (and have been observed)to be uniformly distributed over the whole N -dimensional space [10]. The RA is launchedin such an instance 800 times starting from different random conditions. The histogram

of the Hamming distances d(S1, S2) = 12(1 − (1/N)

∑i x

(S1)i x

(S2)i ) between any two of the

800 solutions peaks at 0.4995 (with a standard deviation of 0.0118).There is no guarantee that the present algorithm is able to address all clusters of

solutions, but the fact that the distance distribution among the found solutions of abar-balanced formula peaks at 0.5 with a standard deviation expected from a randomdistribution (i.e. 2/

√N = 0.0115) suggests that the addressable solutions could be

homogeneously distributed over the space of the clusters of solutions.

5.3.4. Logarithmic dependence of time on the size of the graph. One now analyses theevolution of the convergence time as a function of the size of the graph. For this, onedetermines, for a fixed value of α = 4.22, the simulated time that the convergence wouldtake if implemented on a distributed device, as a function of the size N of the graph(figure 9). For N ranging from 9000 to 9×105, the time of convergence is well approximatedby a logarithmic fit (t = −315+67∗ log(N)). The unit of time is one parallel update of allη. Each point corresponds to a different random formula. Note that, above N > 3×104, alltrials were successful, and the RA invariably leads to a satisfying assignment. For smallergraphs, due to finite-size effects, the algorithm either fails to converge or converges to anassignment with a few contradictions (generally one to four) in a few cases: at most 5%of the experiments for N = 9000 and none for N > 30 000.

As local messages need a logarithmic time to reach all parts of the graph, and as, inthe hard-SAT phase, the assignment of a value to a given variable i does not depend onpurely local constraints but on the state of the whole graph, one expects that, by the useof purely local messages on a distributed device, the problem cannot be solved faster thanin a logarithmic time.

5.3.5. Performance of the reinforcement algorithm on a serialized implementation. Numericalexperiments show that RA scales as N × log(N) when implemented on a serializedcomputer. In the scheme of the classical SID, a slightly slower scaling, in N × (log(N))2,can be achieved by fixing after each convergence a given percentage of the total amountof variable [11].

Moreover, Braunstein et al [11] reports that, for α = 4.24 and N = 105, by fixing atleast f = 0.125% of the N variables at the end of each convergence, the SID reaches aparamagnetic state after an average of 7460 sweeps, i.e. 7460 updates of all remaining η.Conversely, the synchronous RA only needs 600 parallel updates of the η plus 300 parallelupdates of the forcing messages to converge towards a solution. Its asynchronous versionrequires 450–500 complete updates of the η and of the forcing messages to arrive at asolution. This means that, in a serialized computer, close to the critical point, the presentalgorithm performs faster than the classical SID by a factor of five. This can be checkedby running the two algorithms on a 2.4 GHz PC: the formula (N = 105 and α = 4.24)is solved on average in 30 min using the SID with f = 0.125% and in 7 min using the‘reinforcement algorithm’ (either asynchronous or synchronous).

doi:10.1088/1742-5468/2005/11/P11016 22

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

100

200

300

400

500

600

700

800

900

1000

10000 100000 1e+06

nb s

wee

ps u

ntil

conv

erge

nce

N, size of graph

-315+67*log(N)

Figure 9. The number of parallel sweeps needed to find a solution dependslogarithmically on N (α = 4.22, π = 0.04). The examples in which theconvergence fails (due to finite size effects, which have not been observed forN > 30 000) are not represented.

We finally characterize the slowing down of the RA when approaching the criticalvalue αC = 4.2667 (figure 10). Far away from the critical point, the time of convergenceis highly reliable. Close to the critical point, the algorithm gets slower and the time ofconvergence becomes more variable. Setting the intensity of the forcing at its optimalvalue, one finds that, for α < αA, the time of convergence is inversely proportional to thecomplexity Σ: t = tΣ0 /Σ (tΣ0 = 1.474, figure 10, left panel); moreover, as expected fromthe pseudo-linear relationship between Σ and α, the time of convergence also scales with1/(α − αC): t = tα0 /(α − αC) (tα0 = 3.02, figure 10, right panel).

Very close to αC (α > αA), the RA is unable to find ground states and only exploresquasi-optimal assignments.

6. Conclusion

This study aimed at making fully distributed the survey propagation decimationalgorithm.

The first distributed solver, the ‘SP diffusion algorithm’, diffuses as dynamicalinformation the maximum bias over the system, so that variable nodes can decide to freezein a self-organized way. Its properties (solving rate, final number of active clauses when itreaches the paramagnetic state) are comparable with the previously described serialized‘SP inspired decimation’. The new feature is that the simulated time of convergencetowards a solution, if this was implemented on a fully distributed device, goes as log(N),i.e. scales optimally with the size of the graph.

doi:10.1088/1742-5468/2005/11/P11016 23

J.Stat.M

ech.(2005)P

11016

Survey-propagation decimation through distributed local computations

1000

300

100

50

0.01 0.003 0.001

t

Σ

fit 1000

300

100

50

0.1 0.03 0.01

t

fit

1-α/αC

Figure 10. Time of convergence as a function of Σ (left) or α (right), using forπ the optimal intensity of the forcing (N = 105). Close to the critical point,the time of convergence is shown to be inversely proportional either to Σ or to1 − α/αC. The fit of the data t = tΣ0 × Σ−β over Σ ∈ [0, 0.008] gives tΣ0 = 1.474and β = 0.9994. The fit t = tα0 ×(1−α/αC)−γ over α ∈ [4.14, 4.25] gives tα0 = 3.02and γ = 1.057.

The second solver, the ‘SP reinforcement algorithm’, makes use of time-dependentlocal external forcing messages, which let the variables get completely polarized in thedirection of a solution at the end of a single convergence. The estimated time ofconvergence of this solver, when implemented on a distributed device, also goes as log(N).

The present algorithm is the fastest existing solver of the K-SAT formulae, whenimplemented either on a distributed device or on a serialized computer.

Moreover, the strategies proposed in the present manuscript are not specific to the K-SAT problem, but are likely to apply to many types of NP-complete constraint satisfactionproblem (CSP) with direct applications (e.g. data compression algorithm [1]).

The different codes are available at [22, 23].

Acknowledgments

This work was supported by EVERGROW, integrated project No 1935 in the complexsystems initiative of the Future and Emerging Technologies Directorate of the IST Priority,EU Sixth Framework. We thank Demian Battaglia and Alfredo Braunstein for very helpfuldiscussions.

References

[1] Ciliberti S, Mezard M and Zecchina R, Lossy data compression with random gates, 2005 Phys. Rev. Lett.95 038701 [cond-mat/0504509]

[2] Cocco S, Dubois O, Mandler J and Monasson R, Rigorous decimation-based construction of ground purestates for spin glass models on random lattices, 2003 Phys. Rev. Lett. 90 047205

doi:10.1088/1742-5468/2005/11/P11016 24

J.Stat.M

ech.(2005)

P11016

Survey-propagation decimation through distributed local computations

[3] Dubois O, Monasson R, Selman B and Zecchina R (ed), Phase transitions in combinatorial problems, 2001Theor. Comput. Sci. 265 1

[4] Mezard M, Mora T and Zecchina R, Clustering of solutions in the random satisfiability problem, 2005 Phys.Rev. Lett. 94 197205 [cond-mat/0504070]

[5] Mezard M, Ricci-Tersenghi F and Zecchina R, Two solutions to diluted p-spin models and XORSATproblems, 2003 J. Stat. Phys. 111 505

[6] Achlioptas D and Ricci-Tersenghi F, 2005 in preparation[7] Richardson T and Urbanke R, The capacity of low-density parity-check codes under message-passing

decoding , 2001 IEEE Trans. Inf. Theory 47 599[8] Luby M G, Mitzenmacher M, Shokrollahi M A and Spielman D A, 2001 IEEE Trans. Inf. Theory 47 585[9] Battaglia D, Braunstein A, Chavas J and Zecchina R, 2005 Phys. Rev. E 72 015103

[10] Battaglia D, Braunstein A, Chavas J and Zecchina R, 2005 Prog. Theor. Phys. Suppl. 157 330[11] Braunstein A, Mezard M and Zecchina R, 2005 Random Struct. Algorithms 27 201[12] Gallager R G, Low density parity check codes, 1962 IRE Trans. Inf. Theory 8 21[13] Kschischang F R, Frey B J and Loeliger H-A, Factor graph and the sum–product algorithm, 2001 IEEE

Trans. Inf. Theory 47 498[14] Levine B, Taylor R R and Schmit H, Implementation of near Shannon limit error-correcting codes using

reconfigurable hardware, 2000 Proc. IEEE Symp. on Field Programmable Computing Machines pp 217–26[15] MacKay D J C, Good error-correcting codes based on very sparse matrices, 1999 IEEE Trans. Inf. Theory

45 399[16] Mezard M, Parisi G and Zecchina R, Analytic and algorithmic solution of random satisfiability problems,

2002 Science 297 812[17] Mezard M and Zecchina R, The Random K-satisfiability problem: from an analytic solution to an efficient

algorithm, 2002 Phys. Rev. E 66 056126[18] Selman B, Kautz H and Cohen B, 1994 Proc. AAAI-94 (Seattle, WA) pp 337–43[19] Parisi G, Some remarks on the survey decimation algorithm for K-satisfiability , 2003 Preprint

cs.CC/0301015[20] Parisi G, A backtracking survey-propagation algorithm for K-satisfiability , 2003 Preprint cond-mat/0308510[21] Yedidia J S, Freeman W T and Weiss Y, Generalized belief propagation, 2001 Advances in Neural

Information Processing Systems vol 13 (Cambridge, MA: MIT Press) pp 689–95[22] The diffusion code: http://www-rocq.inria.fr/who/Cyril.Furtlehner/[23] The asynchronous and the synchronous reinforcement code: http://www.ictp.trieste.it/∼zecchina/SP/

doi:10.1088/1742-5468/2005/11/P11016 25