Noname manuscript No.(will be inserted by the editor)

Hyper-heuristics: A Survey of the State of the Art

Edmund K. Burke · Matthew Hyde ·Graham Kendall · Gabriela Ochoa · Ender

Ozcan · Rong Qu

Received: date / Accepted: date

Abstract Hyper-heuristics comprise a set of approaches which are motivated (at least

in part) by the goal of automating the design of heuristic methods to solve hard compu-

tational search problems. An underlying strategic research challenge is to develop more

generally applicable search methodologies. The term hyper-heuristic was first used in

1997 to describe a protocol that combines several Artificial Intelligence methods in the

context of automated theorem proving. The term was independently used in 2000 to

describe ‘heuristics to choose heuristics’ in the context of combinatorial optimisation.

However, the idea of automating the design of heuristics is not new. It can be traced

back to the 1960s and 1970s.

We distinguish between two main hyper-heuristic categories: heuristic selection and

heuristic generation. The distinguishing feature of hyper-heuristics is that they operate

on a search space of heuristics (or heuristic components) rather than directly on the

search space of solutions to the underlying problem that is being addressed. This paper

presents a critical discussion of the scientific literature on hyper-heuristics including

their origin and intellectual roots, a detailed account of the main types of approaches,

and an overview of some related areas. Current research trends and directions for future

research are also discussed.

Keywords hyper-heuristics · combinatorial optimisation · search methodologies

1 Introduction

Despite the success of heuristic methods and other search techniques in solving real-

world computational search problems, there are still some difficulties in terms of easily

applying them to newly encountered problems, or even new instances of similar prob-

lems. These difficulties arise mainly from the significant range of parameter or algorithm

choices involved when using this type of approach and the lack of guidance as to how

to select them. In addition, the scientific community’s level of understanding of why

Automated Scheduling, Optimisation and Planning (ASAP) research groupUniversity of Nottingham, School of Computer Science,Jubilee Campus, Wollaton Road, Nottingham NG8 1BB, UKE-mail: {ekb,mvh,gxk,gxo,exo,rxq}@cs.nott.ac.uk

2

and how different heuristics work effectively (or not) in different situations does not

facilitate simple choices of which approach to use in which situation. Another draw-

back of current techniques is that state-of-the-art approaches for real-world problems

tend to represent bespoke problem-specific methods which are expensive to develop

and maintain. A key motivating goal for this area (but by no means the only one) is

the challenge of automating the design and tuning of heuristic methods to solve hard

computational search problems (Burke et al, 2003a; Ross, 2005; Burke et al, 2009a).

The key idea is to develop algorithms that are more generally applicable than many of

the current implementations of search methodologies. When using hyper-heuristics, we

are attempting to find the right method or sequence of heuristics in a given situation

rather than trying to solve the problem directly. Hyper-heuristics could be regarded as

“off-the-peg” methods as opposed to “made-to-measure” techniques. Therefore, an im-

portant goal is to design generic methods, which should produce solutions of acceptable

quality, based on a set of easy-to-implement low-level heuristics. A hyper-heuristic can

be seen as a (high-level) methodology which, when given a particular problem instance

or class of instances, and a number of low-level heuristics (or its components), auto-

matically produces an adequate combination of the provided components to effectively

solve the given problem(s).

We can identify two fundamental ideas behind the notion of hyper-heuristics.

Firstly, the recognition that the process of selecting or designing efficient hybrid and/or

cooperative heuristics can be regarded as a computational search problem in itself. Sec-

ondly, there is significant potential to improve search methodologies by the incorpora-

tion of learning mechanisms that can adaptively guide the search. These two funda-

mental ideas have inspired different types of hyper-heuristics (Burke et al, 2009b). The

term hyper-heuristic was first used in 1997 (Denzinger et al, 1997) to describe a proto-

col that combines several Artificial Intelligence methods in the context of automated

theorem proving. The term was independently used in 2000 (Cowling et al, 2000) to

describe ‘heuristics to choose heuristics’ in the context of combinatorial optimisation.

The first journal paper to use the term was (Burke et al, 2003b). In this context a

hyper-heuristic is a high-level approach that, given a particular problem instance and

a number of low-level heuristics, can select and apply an appropriate low-level heuristic

at each decision point. The idea of automating the design of heuristics, however, is not

new. It can be traced back to the early 1960s and 1970s, as we will discuss in section

2.

A number of introductory tutorial and review book chapters on hyper-heuristics

have been published over the last few years. The first one appeared in 2003 (Burke

et al, 2003a), where the authors discussed the idea of hyper-heuristics and stressed

one of the key objectives; namely, to raise the level of generality at which optimi-

sation systems can operate. The chapter also gives a brief history of the area and

discusses in detail some representative examples published at the time. A tutorial ar-

ticle was later published by Ross (2005), which not only gives useful guidelines for

implementing a hyper-heuristic approach; but it also discusses a number of relevant

research issues and identifies promising application domains. A more recent publication

(Chakhlevitch and Cowling, 2008) provides a classification and discussion of recent de-

velopments in hyper-heuristics. The chapter, however, does not consider the heuristic

generation methodologies, nor does it include a detailed account of the origins and early

hyper-heuristic approaches. A recent overview and tutorial chapter (Burke et al, 2009a)

discusses methodologies to generate new heuristics from a set of potential heuristic com-

ponents, in which Genetic Programming plays a prominent role. The chapter includes

3

a detailed description of the steps needed to apply this approach, some representative

case studies, a brief literature review of related work, and a discussion of relevant is-

sues of this class of hyper-heuristic. Finally, Burke et al (2009b) present an overview

of previous categorisations of hyper-heuristics and provide a unified classification and

definition which captures the work that is being undertaken in this field. Two main

hyper-heuristic categories are identified: heuristic selection and heuristic generation,

and some representative examples of each category are discussed in detail.

The remainder of this article is structured as follows. The next section discusses

the intellectual roots and early hyper-heuristic approaches. We consider as early ap-

proaches, those that appeared before the year 2000, when the term hyper-heuristic

started to be widely used to refer to the idea of “heuristics to choose heuristics”. Section

3 discusses our proposal for classifying hyper-heuristics (Burke et al, 2009b). Following

this classification, we then provide a critical discussion of the modern scientific liter-

ature covering heuristic selection methodologies (Section 4), and heuristic generation

methodologies (Section 5). Section 6 briefly overviews some related approaches that

also seek to automate the design and tuning of search algorithms. Finally, section 7

highlights the main research trends in hyper-heuristics and suggests some potentially

interesting future research directions.

2 Origins and early approaches

The ideas behind hyper-heuristics are not new. They can be traced back to the early

1960s and 1970s, and can be found within different fields of computer science, artificial

intelligence and operational research. We describe below some relevant intellectual

roots and early approaches developed before 2000, when the term started to be widely

used in relation to the idea of automating the design of heuristics. We have identified

the five following types of early approaches:

2.1 Automated heuristic sequencing

Fisher and Thompson (1961, 1963) and Crowston et al (1963), hypothesised that com-

bining scheduling rules (also known as priority or dispatching rules) in production

scheduling would be superior to using any of the rules separately. This pioneering work,

well ahead its time, proposed a method of combining scheduling rules using ‘probabilis-

tic learning’. Notice that in the early 1960s, computational search methodologies were

far from mature. The term ‘metaheuristic’ (coined by Glover (1986)) was many years

away and local search methods were relatively unsophisticated compared to today’s

methods. However, the learning approach proposed in (Fisher and Thompson, 1961,

1963; Crowston et al, 1963) resembles a stochastic local search algorithm operating in

the space of scheduling rules’ sequences. The main conclusions from this study are the

following: “(1) an unbiased random combination of scheduling rules is better than any

of them taken separately; (2) learning is possible” (Fisher and Thompson, 1963). The

emergence, about forty years later, of a growing interest in hyper-heuristic methods

(Dorndorf and Pesch, 1995; Fang et al, 1993; Storer et al, 1992, 1995), has been facil-

itated by the emergence of appropriate algorithmic tools and computational power to

thoroughly investigate and explore search within a heuristic search space.

4

Storer et al (1992, 1995) clearly stated the problem of designing a good combination

of problem-specific (fast) heuristics as a search problem, and defined neighbourhoods

within the heuristic space. The authors also proposed neighbourhoods on the problem

data. Therefore, by perturbing the heuristic combination, the problem data, or both,

a subset of solutions in these new search spaces were produced. These neighborhoods

formed the basis for local search. The authors stressed that, due to their generic na-

ture, the basic principle can be applied to develop heuristics for a broad spectrum of

problems. Since the emphasis is placed on developing search spaces, the authors used

a simple variant of hill-climbing. They stressed, however, that various search strategies

such as simulated annealing, genetic algorithms or tabu search could be applied. The

ideas were successfully tested by developing heuristics for job-shop scheduling. The

resulting heuristics produced encouraging solutions when compared to the state-of-art

shifting bottleneck procedure.

Fang et al (1993, 1994), used a genetic algorithm that searched the space of se-

quences of heuristic choices in the context of open-shop scheduling. A heuristic choice

was represented as a pair (j, h) indicating an uncompleted job j, and a heuristic h

(from a set of scheduling rules) to select a task from that job (j) to insert next into

the schedule. The sequence of pairs is interpreted from left to right to construct a

complete schedule. The approach produced very good results on benchmark problems

including some best-known solutions at that time. Hart and Ross (1998) successfully

applied a variant of this idea to dynamic job-shop scheduling problems. The approach

is based upon evolving combined iterations of previously proposed heuristic algorithms

for choosing tasks to schedule. Very good results were obtained on benchmark problems

when tested on a variety of criteria. Hart et al (1998) also used a genetic algorithm

based approach to solve a real-world scheduling problem. The problem was to schedule

the collection and delivery of live chickens from farms all over Scotland and Northern

England to one of two processing factories, in order to satisfy a set of orders. The

approach combined two genetic algorithms that evolved heuristic choices, one to man-

age the assignment of orders, and the second to schedule the arrival of deliveries. The

proposed strategy successfully met the project’s requirements.

Norenkov and Goodman (1997) also used evolutionary algorithms for searching

over a space of heuristic sequences. They considered the multistage flow-shop schedul-

ing problem, and included two types of scheduling rules: (i) for determining job order-

ings, and (ii) for determining job assignments to machines. A set of experiments was

conducted and the quality of the solutions obtained was found to strongly depend on

the subset of heuristics applied.

Notice that the approaches discussed above were all ‘online’. That is, directed to

find good sequences of heuristics to solve a given instance of a problem. Also the prob-

lem studied was, in all cases, related to machine scheduling. In contrast the work by

Drechsler and Becker (1995), use an evolutionary algorithm to learn (from a set of

previous examples) successful heuristics that can be applied to instances from a given

problem after a learning phase. This work was developed in the domain of chip design.

In (Drechsler and Becker, 1995), a first application to 2-level AND/EXOR minimi-

sation is presented. A set of Basic Optimization Modules is given and a particular

heuristic is a sequence of these modules. This work was extended in Drechsler et al

(1996), where the learning of heuristics is studied for ordering variables of Binary de-

cision Diagrams. This work is further extended by using genetic programming, instead

of standard evolutionary algorithms as will be discussed in section 5.

5

2.2 Automated planning systems

Another body of work that inspired the concept of hyper-heuristics came from the

Artificial Intelligence community. In particular, from work on automated planning sys-

tems, and its eventual focus towards the problem of learning control knowledge. In

(Gratch and Chien, 1996; Gratch et al, 1993), the so-called COMPOSER system was

used for controlling satellite communication schedules involving a number of earth-

orbiting satellites and three ground stations. The system can be characterised as a

hill-climbing search in the space of possible control strategies. The learning system

alternately proposes changes to the current control strategy and statistically evaluates

them to determine whether they enhance the expected utility. The approach is off-line

in that a large supply of representative training problems are required in order to have

an adequate estimate of the expected utility for various control strategies.

2.3 Automated parameter control in evolutionary algorithms

Some early approaches to automatically set the parameters of evolutionary algorithms

can also be considered as antecedents of hyper-heuristics. Several mechanisms for

modifying parameters during the run in an “informed” way were proposed early in

the history of evolutionary algorithms (Eiben et al, 1999). Examples are the use of

Rechenberg’s 1/5 success rule for changing the mutation rate in evolution strategies

(Rechenberg, 1973b), and Davis’ exploration of changing the crossover rate within ge-

netic algorithms based on the progress of particular crossover operators (Davis, 1989).

Other early proposals for adapting the operator probabilities can be found in (Jul-

strom, 1995; Lobo and Goldberg, 1997; Smith and Fogarty, 1997; Tuson and Ross,

1998). Another idea is that of using an evolutionary algorithm to tune an evolution-

ary algorithm. This can be done using two evolutionary algorithms: one for problem

solving and another one (so-called meta-evolutionary algorithm) to tune the first one

(Freisleben and Hartfelder, 1993; Grefenstette, 1986). It can also be done by using

a single evolutionary algorithm that tunes itself to a given problem while solving it.

The notion of ‘self-adaptation’, first introduced within evolution strategies for varying

the mutation parameters (Rechenberg, 1973a; Schwefel, 1977), is an example in this

category. Self-adaptation in evolutionary algorithms means that some parameters are

varied during a run in a specific manner: the parameters are included in the chro-

mosomes and co-evolve with the solutions. These approaches are related to the idea

of searching over a space of possible algorithm configurations, and are, therefore, re-

lated to hyper-heuristics. For an overview and classification of approaches to parameter

control in evolutionary algorithms, the reader is referred to (Eiben et al, 1999, 2007).

2.4 Automated learning of heuristic methods

An early approach to the automated generation of heuristic computer programs can

be found in the domain of constraint satisfaction problems (Minton, 1996). Specifi-

cally the pioneering work of Minton (1996) presents a system for generating reusable

heuristics for Minimum Maximal Matching Problem. The system modifies given ele-

ments of algorithm ‘schema’, which are templates of generic algorithms. The general

idea is to automatically synthesize problem-specific versions of constraint satisfaction

6

algorithms. The proposed framework “actively searches the space of heuristic algo-

rithms, so that the configuration process itself is a complex and interesting search

process” (Minton, 1996). The user provides a set of training instances, that the system

can experiment with during the configuration processes. In this sense, the system is,

as the author suggests, auto-configurable. It configures itself for a particular problem,

indeed to the particular instance distribution represented by the training instances.

This study is unique in the literature, as the automated approach is compared against

those produced by three NASA programmers, to assess whether the automatically gen-

erated heuristics are human competitive. Both the programmers and the automatical

approach were given a maximum of eight hours to develop their heuristics. The results

were very competitive with the human generated heuristics, and better in one out of

the three experiments

Another interesting learning approach in the mid 1990s was termed “Teacher”

(Wah et al, 1995; Wah and Ieumwananonthachai, 1999) (an acronym for TEchniques

for the Automated Creation of HEuRistics), which was designed as a system for learn-

ing and generalising heuristics used in problem solving. The objective was to find, under

resource constraints, improved heuristic methods as compared to existing ones, in ap-

plications with little or non-existent domain knowledge. The Teacher system employed

a genetic-based machine learning approach, and was successfully applied to several do-

mains such as: process mapping, load balancing on a network of workstations, circuit

placement, routing and testing, among others.

2.5 Automated prioritising: “Squeaky Wheel” optimisation

The general approach to optimisation termed “Squeaky Wheel” (Joslin and Clements,

1999), can be considered as an antecedent to hyper-heuristics in that it finds solutions

quickly by operating on two search spaces: the traditional solution space and a new

priority space. In squeaky wheel, a priority ordering of solution components is given

to a greedy constructive algorithm. The constructed solution is then analysed to find

trouble spots (i.e. components, that if improved, are likely to improve the objective

function value). The priority of these trouble spots is increased, and the new priority

ordering is given to the greedy constructor. This construct/analyze/prioritize cycle

continues until a predefined criteria is met. Given that squeaky wheel operates on

a dual search space (solutions and priorities), it avoids some of the problems that a

standard local search often encounters. Successive solutions in squeaky wheel are only

indirectly related, via the recalculation of priorities. In consequence, the algorithm can

effectively make large coherent ‘jumps’ to quickly move out of unpromising regions of

the solution search space. The authors suggest that this ability to make large moves

is a strength of the approach, but it is also a weakness. Making small “tuning” moves

is not easy, but the coupled-search view of squeaky wheel suggests an obvious remedy:

its combination with local search in the solution space.

3 A classification of hyper-heuristic approaches

In Burke et al (2009b), a general classification of hyper-heuristics is proposed, according

to two dimensions : (i) the nature of the heuristic search space, and (ii) the source of

feedback during learning. These dimensions are orthogonal in that different heuristic

7

search spaces can be combined with different sources of feedback, and thus different

machine learning techniques. This classification (summarised in Figure 1) serves as a

framework for our literature survey.

Methodologies to select

Methodologies to generate

constructive

heuristics

perturbative

heuristics

constructive

heuristics

perturbative

heuristics

Online

learning

Offline

learning

No-

learning

Feedback Nature of the heuristic search space

Hyper-

heuristics

Heuristic selection

Heuristic generation

Fig. 1 A classification of hyper-heuristic approaches, according to two dimensions (i) thenature of the heuristic search space, and (ii) the source of feedback during learning.

We consider that the most fundamental hyper-heuristic categories are those repre-

sented by the processes of:

– Heuristic selection: methodologies for choosing or selecting existing heuristics

– Heuristic generation: methodologies for generating new heuristics from the com-

ponents of existing ones

There is no reason why the higher level strategy (for selecting or generating heuris-

tics) should be restricted to be a heuristic. Indeed, sophisticated knowledge-based tech-

niques such as case-based reasoning has been employed in this way with good results

for university timetabling (Burke et al, 2006b). This leads us to propose the following

more general definition of the term ‘hyper-heuristic’ which is intended to capture the

idea of a method for automating the heuristic design and selection process:

A hyper-heuristic is a search method or learning mechanism for selecting or

generating heuristics to solve computational search problems.

From this definition, there are two clear categories of hyper-heuristics: heuristic se-

lection and heuristic generation, which form the first branch in our first dimension (the

nature of the search space). The second level in this dimension corresponds to the dis-

tinction between constructive and local search (perturbative-based) hyper-heuristics.

Notice that this last categorisation is concerned with the nature of the low-level heuris-

tics used in the hyper-heuristic framework, and therefore with how the search in the

solution space is conducted. Our classification uses the terms constructive and per-

turbative to refer to these classes. Sections 4 and 5 describe these categories in more

detail.

We consider a hyper-heuristic to be a learning algorithm when it uses some feedback

from the search process. Therefore, non-learning hyper-heuristics are those that do not

8

use any feedback. According to the source of the feedback during learning, we propose

a distinction between online and offline learning. Notice that in the context of heuristic

generation methodologies, an example of which is genetic programming-based hyper-

heuristics, the notions of disposable and reusable have been used to refer to analogous

ideas to those of online and offline learning, as described below:

Online learning hyper-heuristics: The learning takes place while the algorithm is

solving an instance of a problem. Therefore, task-dependent local properties can be

used by the high-level strategy to determine the appropriate low-level heuristic to

apply. Examples of online learning approaches within hyper-heuristics are: the use

of reinforcement for heuristic selection and, generally, the use of meta-heuristics as

high-level search strategies across a search space of heuristics.

Offline learning hyper-heuristics: The idea is to gather knowledge in the form

of rules or programs, from a set of training instances, that will hopefully gener-

alise to solving unseen instances. Examples of offline learning approaches within

hyper-heuristics are: learning classifier systems, case-base reasoning and genetic

programming.

The proposed classification of hyper-heuristic approaches can be summarised as

follows (see also figure 1):

– With respect to the nature of the heuristic search space

– Heuristic selection methodologies: Produce combinations of pre-existing:

• Constructive heuristics

• Perturbative heuristics

– Heuristic generation methodologies: Generate new heuristic methods us-

ing basic components (building-blocks) of:

• Constructive heuristics

• Perturbative heuristics

– With respect to the source of feedback during learning

– Online learning hyper-heuristics: Learn while solving a given instance of a

problem.

– Offline learning hyper-heuristics: Learn, from a set of training instances,

a method that would generalise to unseen instances.

– Hyper-heuristics without learning: Do not use feedback from the search

process.

Note that these categories describe current research trends. There is, however,

nothing to stop the exploration of hybrid methodologies that combine for example

constructive with perturbative heuristics, or heuristic selection with heuristic genera-

tion methodologies. These hybrid approaches are already starting to emerge.

4 Modern heuristic selection methodologies

4.1 Approaches based on constructive low-level heuristics

These approaches build a solution incrementally. Starting with an empty solution,

they intelligently select and use constructive heuristics to gradually build a complete

solution. The hyper-heuristic framework is provided with a set of pre-existing (generally

9

problem specific) constructive heuristics, and the challenge is to select the heuristic that

is somehow the most suitable for the current problem state. This process continues until

the final state (a complete solution) has been reached. Notice that there is a natural

ending to the construction process when a complete solution is reached. Therefore,

the sequence of heuristic choices is finite and determined by the size of the underlying

combinatorial problem. Furthermore, there is the interesting possibility of learning

associations between partial solution stages and adequate heuristics for those stages.

Application Domain Reference(s)

Production Scheduling

(Fisher and Thompson, 1961, 1963)

(Storer et al, 1992, 1995)

(Dorndorf and Pesch, 1995)

(Fang et al, 1993, 1994)

(Norenkov and Goodman, 1997)

(Hart and Ross, 1998; Hart et al, 1998)

(Vazquez-Rodrıguez et al, 2007a,b; Ochoa et al, 2009b)

(Vazquez-Rodrıguez and Petrovic, 2010)

(Cano-Belman and and J. Bautista, 2010)

Educational Timetabling

(Terashima-Marın et al, 1999)

(Ahmadi et al, 2003; Asmuni et al, 2005)

(Ross et al, 2004; Ross and Marın-Blazquez, 2005)

(Burke et al, 2005a, 2006b)

(Burke et al, 2007c; Qu and Burke, 2009; Ochoa et al, 2009a)

(Pillay and Banzhaf, 2007; Pillay, 2008)

1D Packing(Ross et al, 2002, 2003; Marın-Blazquez and Schulenburg, 2007)

(Marın-Blazquez and Schulenburg, 2007)

2D Cutting and Packing(Terashima-Marın et al, 2006, 2007, 2008a)

(Garrido and Riff, 2007a,b)

Constraint Satisfaction (Terashima-Marın et al, 2008b)

Vehicle Routing (Garrido and Castro, 2009; Garrido and Riff, 2010)

Table 1 Application domains of heuristic selection methodologies based on constructive low-level heuristics.

Several approaches have recently been proposed to generate efficient hybridisations

of existing constructive heuristics in domains such as timetabling, cutting and packing,

production scheduling, constraint satisfaction, and vehicle routing problems (see table

1). Both online and offline approaches, and different high-level strategies have been

investigated (see table 2). The following subsections survey the approaches according

to the type of low-level heuristics employed, which is in turn related to the application

domain.

4.1.1 Graph-colouring heuristics in timetabling (online approaches)

There is a well known analogy between a basic version of a timetabling problem and

the graph colouring problem. Nodes can represent events and edges represent conflicts

between events. Using this analogy, some timetabling algorithms in the literature are

based upon graph colouring heuristics. Early approaches using evolutionary algorithms

10

High-level strategy Reference(s)

Hill-climbing

(Storer et al, 1992, 1995)

(Gratch and Chien, 1996; Gratch et al, 1993)

(Garrido and Castro, 2009)

Genetic Algorithms

(Dorndorf and Pesch, 1995)

(Fang et al, 1993, 1994)

(Norenkov and Goodman, 1997)

(Hart and Ross, 1998; Hart et al, 1998)

(Terashima-Marın et al, 1999)

(Ahmadi et al, 2003)

(Vazquez-Rodrıguez et al, 2007a,b; Ochoa et al, 2009b)

(Garrido and Riff, 2007a,b, 2010)

(Pillay and Banzhaf, 2007; Pillay, 2008)

Meta-heuristics (Ahmadi et al, 2003)

(TS, ILS, VNS) (Burke et al, 2007c; Qu and Burke, 2009)

Fuzzy Systems (Asmuni et al, 2005)

Scatter Search (Cano-Belman and and J. Bautista, 2010)

Case-based Reasoning (Burke et al, 2005a, 2006b)

(Offline)

Classifier Systems (Ross et al, 2002; Marın-Blazquez and Schulenburg, 2007)

(Offline) (Terashima-Marın et al, 2007)

Messy Genetic Algorithms (Ross et al, 2003, 2004; Ross and Marın-Blazquez, 2005)

(Offline) (Terashima-Marın et al, 2006, 2007, 2008a,b)

Table 2 High-level strategies in heuristic selection methodologies based on constructive low-level heuristics. The strategies that incorporate a training phase (offline learning) are indicatedwith the word ‘Offline’ between brackets. TS, ILS, and VNS, refer to tabu search, iterated localsearch and variable neighbourhood search, respectively.

to evolve instructions for constructing an examination timetable rather than inducing

the actual timetable were proposed in (Ross et al, 1997; Terashima-Marın et al, 1999).

The idea was to use a non-direct chromosome representation based on evolving the

configuration of constraint satisfaction methods for examination timetabling problems.

Ahmadi et al (2003) use a variable neighborhood search algorithm to find good com-

binations of parameterised heuristics in examination timetabling. Several constructive

heuristics are proposed based on a weighted decision function and several basic graph

colouring heuristics. Low-level heuristics are used for exam selection (6 heuristics), pe-

riod selection (2 heuristics), and room selection (3 heuristics). These sets of heuristics

include in the three cases a randomised selection. The approach is tested on a real data

set from the University of Nottingham, UK. No comparisons with other approaches,

or further applications to other data sets, are provided.

Asmuni et al (2005) investigate the potential of implementing a fuzzy system in solv-

ing course timetabling problems. The events (courses) to be scheduled are ordered by

combining the following three heuristics: largest-degree, saturation-degree, and largest-

enrollment. The fuzzy weight of an event is used to represent how difficult it is to

schedule. A restricted form of exhaustive search is used to find the most appropriate

shape for the fuzzy membership functions. The events are decreasingly ordered accord-

ing to their weights, and then they are sequentially chosen to be assigned to the last

11

slot with the least penalty cost value. The algorithm was tested on 11 benchmark data

sets of course timetabling problems with encouraging results.

Burke et al (2007c) developed a hyper-heuristic framework that implements five

commonly used graph colouring heuristics: largest-degree, largest-weighted-degree, color-

degree, largest-enrolment and saturation-degree. A random ordering heuristic was also

considered. Tabu search was employed as the high-level search method for producing

good sequences of the low-level heuristics. Each heuristic list produced by the tabu

search algorithm, is evaluated by sequentially using the individual heuristics to order

the unscheduled events, and thus construct a complete timetable. This work also high-

lights the existence of two search spaces in constructive hyper-heuristics (the heuristic

space and the problem solution). The approach was tested on both course and exam

timetabling benchmark instances with competitive results. This graph-based hyper-

heuristic was later extended in (Qu and Burke, 2009) where a formal definition of the

framework is presented. The authors also compare the performance of several high-level

heuristics that operate on the search space of heuristics. Specifically, a steepest descent

method, iterated local search and variable neighborhood search are implemented and

compared to the previously implemented tabu search. The results suggests that the

choice of a particular neighborhood structure on the heuristic space is not crucial to

the performance. Moreover, iterative techniques such as iterated local search and vari-

able neighborhood search, were found more effective for traversing the heuristic search

space than more elaborate meta-heuristics such as tabu search. The authors suggest

that the heuristic search space is likely to be smooth and contain large plateaus (i.e.

areas where different heuristic sequences can produce similar quality solutions). Qu

and Burke (2009) also study hybridisations of their hyper-heuristic framework with

local search operating on the solution space. They found that this strategy greatly

improves the performance of the overall system, making it competitive with state-of-

the-art approaches on the studied benchmark instances. A further study (Ochoa et al,

2009a), uses the notion of fitness landscapes to study the search space of graph colour-

ing heuristics. The study confirms some the observations about the structure of the

heuristic search space discussed in (Qu and Burke, 2009). Specifically, these landscapes

have a high level of neutrality (i.e. the presence of plateaus). Furthermore, although

rugged, they have the encouraging feature of a globally convex or big valley structure,

which indicates that an optimal solution would not be isolated but surrounded by many

local minima.

Pillay and Banzhaf (2007); Pillay (2008) study the performance of evolutionary

algorithms on a similar search space as that discussed above, namely the space of

combinations of graph colouring heuristics for examination timetabling. In the initial

study (Pillay and Banzhaf, 2007), each element of the population is a string of vari-

able length composed of characters each representing one of the following 5 low-level

heuristics: largest-degree, largest-enrollment, largest-weighted degree, saturation-degree,

and highest-cost. The authors report that the system produced feasible examination

timetables with soft constraints within the range of other search methods employed

for this purpose. They also report that the evolutionary system outperformed previous

hyper-heuristics on a number of the tested instances. A follow up study (Pillay, 2008)

explores the effect of different representations on the performance of the evolutionary

hyper-heuristic system. Specifically, three alternative encodings, fixed length, variable

length, and n − times heuristic combinations were tested. The results suggest that

the fixed length encoding does not perform well. The performance of the other two

encodings was found to be problem dependant.

12

4.1.2 Graph-colouring heuristics in timetabling (offline approaches)

Ross et al (2004); Ross and Marın-Blazquez (2005) apply a messy genetic algorithm

(Goldberg et al, 1990) hyper-heuristic based on graph colouring heuristics to both class

and exam timetabling problems. The idea is to learn associations between problem

states and adequate heuristics for timetabling. Specifically, the system tries to discover

a set of labelled points in the space of the problem states. Each label refers to a

heuristic, and the algorithm works by repeatedly finding the nearest labelled point to

the current condition and applies its label until a complete solution is built. Low-level

heuristics are used to select both events and slots. A heuristic chooses an event to

place, and then a slot to place it in. The approach considerers 16 ways of choosing

an event and 28 ways of choosing a slot. Some points are labelled with an event-

picking heuristic and others are labelled with a slot-picking heuristic. The event-picking

heuristics specify an ordering, from which the first available event that does not violate

a hard constraint is chosen. Various different forms of problem-state description and

methods of measuring the fitness were studied. The approach was able to generate fast

and simple problem-solving algorithms that offer good performance over a range of

exam and class timetabling problems.

Burke et al (2005a, 2006b) use a knowledge discovery technique, case-based reason-

ing (Leake, 1996), as a heuristic selector for solving both course and exam timetabling

problems. A set of three graph colouring heuristics: largest-degree, colour-degree, saturation-

degree) and a hill-climbing procedure were selected as low-level heuristics. In (Burke

et al, 2006b), tabu search is employed to discover the most relevant features used in

evaluating the similarity between problem solving situations. The objective was to

choose the best heuristics from the most similar previous problem solving situation to

construct good solutions for the problem in hand. In (Burke et al, 2005a), different ways

of hybridising the low level graph heuristics (with and without case-based reasoning)

were compared for solving the Toronto datasets. It was shown that employing knowl-

edge based techniques rather than randomly/systematically hybridising heuristics in a

hyper-heuristic framework produced better results.

4.1.3 Dispatching rules in production scheduling

Dispatching rules are among the most frequently applied heuristics in production

scheduling, due to their ease of implementation and low time complexity. Whenever a

machine is available, a dispatching rule inspects the waiting jobs and selects the job with

the highest priority to be processed next. Dispatching rules differ from each other in the

way that they calculate priorities. As discussed in section 2, many early hyper-heuristic

approaches were based on dispatching rules. More recently Vazquez-Rodrıguez et al

(2007a) considered combinations of different despatching rules to solve a multi-machine

cardboard box shop scheduling problem. A standard genetic algorithm was employed

as the high level search of sequences of 13 despatching rules: minimum release time,

shortest processing, time, longest processing time, less work remaining, more work re-

maining, earliest due date, latest due date, weighed shortest processing time, weighted

longest processing time, lowest weighted work remaining, highest weighted work remain-

ing, lowest weighted due date and highest weighted due date. The hyper-heuristic was

shown to be capable of learning effective hybridisations of despatching rules during

scheduling, and thus was superior to employing single rules for the whole scheduling

process. A hybrid hyper-heuristic was also developed where a genetic algorithm was

13

employed to solve the first stage of the problem while the rest of the problem is handled

by the hyper-heuristic. Experiments using different objective functions demonstrated

both the effectiveness and generality of this hybrid hyper-heuristic. A multi-objective

job shop problem is studied in (Vazquez-Rodrıguez and Petrovic, 2010) with a similar

approach. Solutions are represented as sequences of dispatching rules that are called

one at a time and used to sequence a number of operations onto machines. This num-

ber of operations is found to be a notoriously sensitive parameter. The hyper-heuristic

in (Vazquez-Rodrıguez and Petrovic, 2010) can be seen as a hybrid method, which

simultaneously searches for the best sequence of rules, and the number of operations to

be handled by each rule. The method uses the selection mechanism of thee well known

NSGA-II (Deb et al, 2000) when handling multi-objective problems. On different vari-

ants of the multi-objective job shop, the method obtained better results on all the

studied instances, when compared with a previous hyper-heuristic based on dispatch-

ing rules, and a conventional genetic algorithm using a permutation representation.

A study of the search space composed by sequences of dispatching rules is presented

in (Vazquez-Rodrıguez et al, 2007b), where a formal definition and some properties of

these spaces are discussed. The notion of a decision block is also introduced to refer to

a set of decisions that are treated as a single unit (i.e. processed by a single heuristic).

A further study (Ochoa et al, 2009b), conducts a landscape analysis of the dispatch-

ing rules search space. Two different objective functions, and several hyper-heuristic

representation sizes (with different block sizes) are considered. The study confirms the

suitability of these heuristic search spaces for evolving solutions to production schedul-

ing problems. Moreover, some characteristics of the scheduling-rules landscapes were

found to resemble those of the graph colouring hyper-heuristic landscapes (Ochoa et al,

2009a), which suggests that constructive heuristic search spaces share some features

that may be exploited by high-level search strategies.

Cano-Belman and and J. Bautista (2010) proposed a a hyper-heuristic embedded

within a scatter search (Laguna and Martı, 2003) framework. The approach is applied

to the problem of sequencing products (mixed-model) on a paced assembly line, and

considers a set of 20 priority rules as low level heuristics. These priority rules are

used to select a product among a set of candidates, and are based on product and

work station features such as demand, processing time, idle time, work overload, etc.

Following the scatter search methodology, the so called reference set contains sequences

of priority rules, whose combination is based on a rule frequency matrix. Two hyper-

heuristic variants were proposed and tested over a wide range of instances from the

literature. The solutions obtained were, in many cases, of better quality than those

found by previous state-of-the art approaches. An interesting aspect of this study, is

that the rule frequency matrix suggests the elimination of some priority rules that do

not contribute significatively to the search process. The rules based on partial work

overload, and partial idle time, were found to be the most relevant.

4.1.4 Packing heuristics in 1D packing problems

Ross et al (2002) used a modern classifier system (accuracy-based classifier system

(Wilson, 1995)), in the domain of one-dimensional bin-packing, to learn a set of rules

that associate characteristics of the current state of a problem with different low-level

constructive heuristics. A simplified description of the current state of the problem

is proposed, which considers the number of items remaining to be packed and their

size ranges. The set of rules evolved by the classifier system is used as follows: given

14

the initial problem characteristics P , a heuristic H is chosen to pack an item, thus

gradually altering the characteristics of the problem that remains to be solved. At each

step, a rule appropriate to the current problem state P ′ is selected, and the process

continues until all items have been packed. For the learning process, a total of 890

benchmark instances from the literature were used. The authors evaluated a variety of

previously proposed bin-packing heuristics, and selected the four that produced the best

results on the studied benchmark set: largest-fit-decreasing, next-fit-decreasing, Djang

and Finch’s (DJD) heuristic, and a variation of DJD. The approach uses single-step

environments, meaning that rewards were available only after each action had taken

place. The classifier system was trained on a set of example problems and showed good

generalisation to unseen problems. In a similar setting, Ross et al (2003) use the messy

genetic algorithm hyper-heuristic (described above in the context of timetabling) for

learning associations between problem states and adequate heuristics for bin-packing.

The approach is applied to the 1D bin-packing problem, and overall the results were

found a little better than those obtained with the classifier system, and on a larger set

of benchmark problems.

Marın-Blazquez and Schulenburg (2007), extended the classifier system approach

to multi-step environments. The authors tested several reward schemes in combination

with alternate exploration/exploitation ratios, and several sizes and types of multi-step

environments. Again, the approach was tested using a large set of benchmark 1D bin

packing problems. The classifier system was able to generalise well and create solution

processes that performed competitively well on a large set of NP-hard benchmark

instances. The authors report that multi-step environments can obtain better results

than the single-step could achieve at the expense of a higher number of training cycles.

4.1.5 Packing heuristics in 2D packing and cutting stock problems

Terashima-Marın et al (2006) applied the messy genetic algorithm hyper-heuristic to

solve 2D regular cutting stock problems. The authors also compared their classifier sys-

tem hyper-heuristic against the messy genetic algorithms approach when solving 2D

bin packing problems, reporting competitive results for both approaches (Terashima-

Marın et al, 2007). For a 2Dl problem, additional difficulty is introduced because it is

necessary to specify where a particular figure should be placed inside the object. There-

fore, two types of heuristics are required: for selecting the figures and objects, and for

placing the figures into the objects. The proposed framework used a set of 8 heuristics

for selecting the items: next-fit, first-fit, best-fit, worst-fit, almost worst fit, first-fit de-

creasing, next-fit-decreasing, and Djang and Fitch heuristics. The placement heuristics

belong to the class of bottom-left heuristics. That is, they keep the bottom-left stability

in the layout. Two heuristics are considered: bottom-left, and improved-bottom-left.

The state of the problem is described by the percentage of pieces that remain to be

packed according to the following categories: huge items, large items, medium items,

and small items. The size of the items is judged in proportion to the object area. A large

empirical study was conducted. The approach produced general hyper-heuristics after

going through a learning process with training and testing phases. When applied to

unseen examples, those hyper-heuristics solved most of the problems very efficiently,

better than the best single heuristic for each instance. A more extensive investiga-

tion of the messy genetic algorithm approach on 2D regular instances is presented in

Terashima-Marın et al (2008a). The study also extended the hyper-heuristic system to

15

handle 2D irregular (convex polygonal) bin packing problems. Very encouraging results

are reported for both types of problems.

Garrido and Riff (2007a,b) also proposed a genetic algorithm based hyper-heuristic

for solving the 2D strip packing problems. In this case the system is online, that is so-

lution methods are evolved for solving a single problem instance. The hyper-heuristic is

based on a set of four low-level heuristics: best-fit, bottom-left-fill, a recursive heuristic

that locates the objects on the bottom left corner (called HR), and a rotational heuris-

tic called BFDH. The approach uses a variable length representation that considers

a categorisation of the low-level heuristics according to their functionality: greedy, or-

dering and rotational. The chromosome also includes the number of objects to be po-

sitioned using each low-level heuristic. The authors compared their approach against

state-of-the-art algorithms on the selected instances. Very good results are reported

that even outperform some of the specialised algorithms for the problem and bench-

mark instances studied.

4.1.6 Other problem domains

Two additional problem domains that have been recently tackled with hyper-heuristics

are constraint satisfaction and vehicle routing.

Terashima-Marın et al (2008b), use the messy genetic algorithms for producing

hyper-heuristics for solving the dynamic variable ordering problem within a constraint

satisfaction framework. When solving constraint satisfaction problems, an important

issue is that of determining which variable is the next to be instantiated. This ordering

has been found to impact the cost of the solution search. The proposed framework

produces combinations of condition-action rules, after going through a learning pro-

cess which includes training and testing phases. The evolved hyper-heuristics produced

encouraging results when tested with large set of randomly generated benchmark prob-

lems.

Garrido and Castro (2009), use a hill-climbing based hyper-heuristic to solve the

capacitated vehicle routing problem. The approach incorporates both constructive and

perturbative heuristics. Specifically, it searches the space of sequences of constructive-

perturbative pairs of low-level heuristics. These sequences are applied in order to con-

struct and improve partial solutions. Four constructive heuristics (Clarke & Wright,

Mole & Jameson, Sweep Algorithm and Kilby), and four perturbative heuristics (2-opt

, 3-opt, Or-opt and Van Breedams moves) are considered. Some ordering heuristics are

also included to arrange the insertion of customers. The approach was tested using

some standard state-of-the-art benchmarks and compared against several well-known

methods proposed in the literature. Stable and good quality solutions were obtained

while solving various types of instances. In a follow up work, the authors use an evo-

lutionary hyper-heuristics for solving hard instances of the dynamic vehicle routing

problem (Garrido and Riff, 2010). The framework includes three types of low-level

heuristics: constructive, perturbative, and noise heuristics, and evolves a sequence of

combinations of them, which are applied in order to construct and improve partial

solutions.The approach is evaluated using the Kilbys benchmarks, which comprise a

large number of instances with different topologies and degrees of dynamism, and com-

pared with some well-known methods proposed in the literature. The hyper-heuristic

was found to adapt to the dynamic nature of the problem and produced high quality

results.

16

4.1.7 Summary and discussion

We have covered the literature on heuristic selection methodologies that are based on

constructive heuristics. Some of the earliest intellectual roots and origins of hyper-

heuristics can be found in this type of approach, specially in the idea of combining

priority or dispatching rules for solving production scheduling rules. From these early

studies it can be inferred that a combination or sequencing of several rules or heuristics

is advantageous over using just a single one. This fact has been confirmed several

times within different domains in recent years. Apart from production scheduling,

which was the first domain studied, the application that has received most attention

is educational timetabling. Specifically, several frameworks have been proposed for

automatically sequencing graph colouring heuristics in educational timetabling. The

experimentation within one of these frameworks Qu and Burke (2009) suggests that

by employing neighborhood moves within the heuristic search space, the high level

search is capable of jumping within the solution space. The heuristic sequences seem

to cover only a subset of the solution space (but well distributed areas). The search

space of heuristics was found to be relatively smooth (i.e. it contains a large number

of plateaus where solutions with the same quality exist). The role of the high-level

heuristic appears to be to search within the limited areas quickly and to explore as

widely as possible the solution space by re-starting from different heuristic sequences

within a limited computational time. This clearly invites to the hybridisation of these

approaches with local search techniques in the solution space as already suggested and

demonstrated in (Qu and Burke, 2009).

Another application domain that has been widely studied within constructive hyper-

heuristics is cutting and packing. Specifically, offline approaches based initially on learn-

ing classifier systems and later on messy genetic algorithms have been largely studied

(Ross et al, 2002, 2003; Marın-Blazquez and Schulenburg, 2007). Firstly, they were

applied to one-dimensional binpacking problems, and later extended to 2D regular and

irregular packing problems (Terashima-Marın et al, 2007, 2008a). One fundamental

research issue in this type of approaches is the determination of a simplified yet ac-

curate representation of the state space in the construction process; since the learning

process is directed to link state-space descriptions to useful low-level heuristics. Other

fundamental issues are the sensitivity of results in relation to the particular choice of

low-level heuristics involved, and whether the use of randomised heuristics is advisable.

The determination of efficient learning or search techniques to be used as high-level

strategies also deserves further study.

More recently, constraint satisfaction problems and vehicle routing have been ad-

dressed with constructive hyper-heuristics (Garrido and Castro, 2009; Garrido and Riff,

2010), (Terashima-Marın et al, 2008b), there is a wide room for improvements in these

new domains. Finally, the exploration of additional domains is another worthwhile

future research direction.

4.2 Approaches based on perturbative low-level heuristics

These approaches aim to improve a candidate solution through a process of automati-

cally selecting and applying a heuristic. Online and offline machine learning techniques

are valuable to the heuristic selection strategies in order to make informed decisions

regarding which heuristic to employ at a given step. Hyper-heuristic methodologies

17

Application domain Reference(s)

Channel assignment Kendall and Mohamad (2004a,b)

Component placement Ayob and Kendall (2003)

Personnel scheduling

Cowling et al (2000)

Cowling and Chakhlevitch (2003)

Han and Kendall (2003)

Burke et al (2003b)

Bai et al (2007a)

PackingDowsland et al (2007)

Bai et al (2007a)

Planning Nareyek (2003)

Production scheduling Ouelhadj and Petrovic (2008, 2009, 2010)

Reactive power compensation Antunes et al (2009)

Space allocation

Burke et al (2005c)

Bai and Kendall (2005)

Bai et al (2008)

Space-probe trajectory optimisation Biazzini et al (2009)

Timetabling

Burke et al (2003b, 2005b)

Bilgin et al (2006)

Chen et al (2007)

Bai et al (2007a)

Ozcan et al (2009)

Vehicle routing problemsPisinger and Ropke (2007)

Meignan et al (2010)

Table 3 Application domains of methodologies to select perturbative heuristics.

based on perturbative heuristics have been applied to a wide variety of combinatorial

optimisation problems as summarised in Table 3.

There are a few studies on the hyper-heuristic methodologies to select perturbative

heuristics that perform multi-point search (processing multiple solutions). The majority

of previously proposed approaches conduct a single point search. In a single point

search based hyper-heuristic framework, an initial candidate solution goes through a

set of successive stages repeatedly until termination. First, a heuristic (or a subset of

heuristics) is selected from a set of low level perturbative heuristics and then applied

to a single candidate solution. Finally, a decision is made whether to accept or reject

the new solution as shown in Figure 2.

A hyper-heuristic to select perturbative heuristics performing single-point search

combines two separate components: (i) heuristic selection method, and (ii) move accep-

tance method as identified in Bilgin et al (2006); Ozcan et al (2008). This component

decomposition presents a high level of modularity, indicating that either one of these

components can be replaced by another method generating a new hyper-heuristic. An

instance of a single-point search based hyper-heuristic will be denoted as Heuristic

Selection–Move Acceptance from this point forward. Different combinations of heuris-

tic selection and move acceptance methods have been explored within the context of

hyper-heuristics.

18

Choose heuristic(s)

Apply chosen

heuristic(s)

Problem Domain

Domain-independent information acquisition and processing: change in a candidate

solution’s quality, number of low level heuristics, measuring the performance of the

applied heuristic, statistics, etc.

Accept / Reject

Low level heuristics

H1

…

Hi

…

Hn

Domain Barier

Representation,

evaluation function,

inititial solution, etc.

Hyper-heuristic

Fig. 2 A hyper-heuristic framework performing single point search

4.2.1 Learning selection in hyper-heuristics performing single point search

The heuristic selection that does not use any type of learning mechanism is based on

either a random or an exhaustive process. A learning mechanism can be introduced

into the heuristic selection process to improve the decision making process over a set

of possible neighbourhoods. Thabtah and Cowling (2008) discusses an offline learning

strategy to detect a rule from seen problem instances to choose a low level heuristic at a

given decision point for solving unseen problem instances. The majority of the heuristic

selection methods used within the single point search based hyper-heuristic framework

generate online score(s) for each heuristic based on their performances. Then these

values are processed and/or combined in a systematic manner to select the heuristic

to be applied to the candidate solution at each step. All score based heuristic selection

techniques require five main components to be implemented: (i) initial scoring, (ii)

memory length adjustment, (iii) strategy for heuristic selection based on the scores,(iv

and v) score update rules in case of improvement and worsening, respectively. All low

level heuristics are assigned an initial score. Depending on the mechanism used, these

scores might affect the performance of a hyper-heuristic. In general, initial scores are set

to the same value, typically zero. Memory length determines the affect of the previous

performance of a heuristic while making the heuristic selection at a decision point.

Given a set of scores, heuristic selection can be performed in many different ways.

For example, max strategy selects the heuristic with the maximal score. On the other

hand, Roulette-wheel (score proportionate) strategy associates a probability with each

heuristic which is computed by dividing each individual score by the total score. Then,

a heuristic is selected randomly based on these probabilities. A high score generates a

higher probability of being selected.

One of the commonly used methods in hyper-heuristics is reinforcement learning,

see Kaelbling et al (1996); Sutton and Barto (1998) for more details. A reinforcement

learning system interacts with the environment (or a model of the environment) and

given a state, takes an action based on a policy. By trial and error, the system at-

19

tempts to learn which actions to perform by evaluating state and action pairs through

accumulated rewards. In the context of hyper-heuristics, rewarding and punishing each

heuristic depending on their individual performance during the search is a scoring

mechanism. If a low-level heuristic improves a solution, then it is rewarded and its

score gets updated positively, while a worsening move causes punishment of a heuristic

by decreasing its score. Different combination of operators can be designed for reward

and punishment.

The acceptance strategy is an important component of any local search heuris-

tic (operating on any search space). Two different types of acceptance strategies can

be identified in the literature: deterministic or non-deterministic. Deterministic meth-

ods make the same decision for acceptance regardless of the decision point during the

search using given current and new candidate solutions(s). A non-deterministic ap-

proach might generate a different decision for the same input. The decision process in

almost all non-deterministic move acceptance methods requires additional parameters,

such as the time (or current iteration). The heuristic selection mechanism in Table 4

summarises well known heuristic selection and move acceptance methods.

Single point search based hyper-heuristics will be covered in four distinct subsec-

tions considering the nature of their components as follows:

– Hyper-heuristics using deterministic move acceptance

– Hyper-heuristics using heuristic selection methods with no learning and non-deterministic

move acceptance

– Hyper-heuristics using heuristic selection methods with learning and non-deterministic

move acceptance

– Comparison studies

4.2.6 presents an overview of multi-point search based hyper-heuristics. Finally, 4.2.7

provides a summary and discussion.

4.2.2 Hyper-heuristics using deterministic move acceptance

In Cowling et al (2000, 2002b), the authors proposed and compared a variety of the

hyper-heuristic components on two real-world scheduling problems: a sales summit

and a project presentation problem, respectively. A Simple Random heuristic selection

method chooses a low level heuristic at random at each step. Random Gradient is a

variant of Simple Random, a randomly selected heuristic is repeatedly applied until

no improvement is achieved. The same affect of Random Gradient can be achieved by

modifying the operation of each heuristic as discussed and employing Simple Random.

Random Permutation generates a random ordering of the low level heuristics and at

each step successively applies a low level heuristic in the provided order. Random Per-

mutation Gradient is a variant of Random Permutation that proceeds in the same man-

ner as Random Gradient without changing the order of heuristics until no improvement

is achieved. Greedy exhaustively applies all low level heuristics to a candidate solution

and selects the one that generates the best improved solution. Although Random Gra-

dient and Random Permutation Gradient heuristic selection methods make use of a

random component, they can still be considered as intelligent heuristic selection mech-

anisms that embed a reinforcement learning mechanism. Initial scores of all heuristics

are set to 0, which is also the lower bound for the scores. The upper bound is set to 1.

As a score update rule, score of an improving heuristic is increased by one (additive),

otherwise it is punished by decreasing its score by 1 (subtractive). The scores are kept

20

Component name Reference(s)

Heuristic selection with no learning

Simple Random Cowling et al (2000, 2002b)

Random Permutation Cowling et al (2000, 2002b)

Greedy Cowling et al (2000, 2002b);

Cowling and Chakhlevitch (2003)

Peckish Cowling and Chakhlevitch (2003)

Heuristic selection with learning

Random Gradient Cowling et al (2000, 2002b)

Random Permutation

Gradient

Cowling et al (2000, 2002b)

Choice Function Cowling et al (2000, 2002b)

Reinforcement Learning Nareyek (2003); Pisinger and Ropke

(2007); Bai et al (2007a)

Reinforcement Learning

with Tabu Search

Burke et al (2003b); Dowsland et al (2007)

Deterministic move acceptance

All Moves Cowling et al (2000, 2002b)

Only Improvements Cowling et al (2000, 2002b)

Improving and Equal Cowling et al (2000, 2002b)

Non-deterministic move acceptance

Monte Carlo Ayob and Kendall (2003)

Great Deluge Kendall and Mohamad (2004a);

Bilgin et al (2006)

Record to Record Travel Kendall and Mohamad (2004b)

Tabu Search Chakhlevitch and Cowling (2005)

Simulated Annealing Bai and Kendall (2005); Bilgin et al

(2006); Pisinger and Ropke (2007); An-

tunes et al (2009)

Simulated Annealing with

Reheating

Dowsland et al (2007); Bai et al (2007a)

Late Acceptance Ozcan et al (2009)

Table 4 Representative studies on heuristic selection and move acceptance strategies (sepa-rated by double lines) in single-point search based hyper-heuristics

within the bounds, hence; the memory length is set to the shortest possible value for

such a reinforcement learning scheme. This type of strategy can be useful if the search

landscape is highly rugged and there are not many plateaus. With the exception of

Greedy, the other heuristic selection methods execute fast.

Choice Function heuristic selection methods introduced in Cowling et al (2000)

are score based techniques that guide the selection of the low-level heuristics during

the search process. The function adaptively ranks the low-level heuristics with respect

to a combined score based on; how well a given heuristic has performed individually

(Equation 1), and as a successor of previously invoked heuristic (Equation 2), and the

elapsed time since the heuristic was last called (Equation 3). The first two components

intensify recent performance, while the third provides an element of diversification. The

parameters α, β (∈ (0, 1]) and δ reflect the relative weight of each component in the

21

overall score for each heuristic (Equation 4). They also utilise a memory. For example,

if a local optimum is achieved and if recently used heuristics do not help, a heuristic

that has not been used for a while is likely to be chosen for invocation. In this way,

the search process is diversified. Scores for the low level heuristics are repeated at each

step. For implementing the heuristic selection, max and roulette wheel strategies were

tested, with the former approach producing better performance.

In(y) and Tn(y) (In(x, y) and Tn(x, y)) denote the change in the evaluation function

and the amount of time taken, respectively, when the nth last time the heuristic y was

selected and employed (immediately after heuristic x).

∀i, g1(hi) =∑n

αn−1 In(hj)

Tn(hj)(1)

∀i, g2(hID, hi) =∑n

βn−1 In(hj , hID)

Tn(hj , hID)(2)

∀i, g3(hi) = elapsedT ime(hi) (3)

∀i, score(hi) = αg1(hi) + βg2(hID, hi) + δg3(hi) (4)

As the acceptance criteria, two deterministic approaches were considered: All Moves

(AM) and Only Improvements (OI). The experimental results in Cowling et al (2000)

show that the Choice Function–All Moves hyper-heuristic is promising. The best pa-

rameter set is obtained through a manual tuning process during the experiments. Cowl-

ing et al (2001) proposes a parameter-less Choice Function that updates all parameters

((α, β and δ)) automatically at each step based on a reinforcement learning scheme.

In Cowling et al (2002b), this parameterless Choice Function–All Moves was found to

outperform the simple ones over the problems studied, and produced improved results

when compared to a manually produced solution and a constructive approach. The

design of this hyper-heuristic is further extended in Rattadilok et al (2005), by propos-

ing a model for general-purpose low-level-heuristics and exploiting parallel computing

frameworks for the hyper-heuristics.

Nareyek (2003) uses Reinforcement Learning (RL) as a heuristic selection method

attempting to learn how to select the promising heuristic at each decision point. The

learning process is based on scores (weights) as described previously. Each heuristic

starts with the same score and they are updated by a predetermined scheme dur-

ing the move acceptance process. The approach is evaluated on the Orc Quest problem

Nareyek (2001), and in a modified Logistics Domain, well known to the action-planning

community. The proposed score update mechanism considers and compares alternative

positive and negative reinforcement update schemes: additive/subtractive, multiplica-

tive/divisional, and power/root adaptation. The heuristic selection process makes a

decision based on the scores of the available heuristics. Once a heuristic is selected it

is directly applied, that is All Moves is used as an acceptance criterion. The results of

the study suggest that combining a low rate of adaptation (additive update) for re-

warding an improvement with a strong (root update) rate of adaptation for punishing

a deterioration is a good choice. Moreover, choosing a heuristic with a max strategy at

each step often generates better results when compared to choosing a heuristic with a

roulette wheel scheme.

Burke et al (2003b) presents Reinforcement Learning with Tabu Search heuristic se-

lection method. Similar to the previous study of Nareyek Nareyek (2003), the low-level

heuristics are selected according to learnt scores (ranks). The proposed hyper-heuristic

22

also incorporates a dynamic tabu list of low-level heuristics that are temporarily ex-

cluded from the available heuristics. The algorithm deterministically selects the low-

level heuristic with the highest rank (max strategy) that is not included in the tabu list.

The chosen heuristic is employed regardless of whether the selected move causes an im-

provement or not (All Moves acceptance). If there is an improvement, the heuristic rank

is increased, otherwise not only the rank is decreased, but also the heuristic is enqueued

into the tabu list. The rank update scheme is additive, and the tabu list is emptied each

time a non-improvement move is accepted. This hyper-heuristic is evaluated on various

instances of two distinct timetabling problems:university course timetabling and nurse

rostering. The results were competitive with those obtained using the state-of-the-art

problem-specific techniques. Burke et al (2005c) extends this methodology with a fixed

size tabu list to be used in multi-objective optimisation. The hyper-heuristic maintains

the scores of low level heuristics for each objective separately. The results show that

the proposed multi-objective hyper-heuristic framework guides the search towards the

promising areas of the trade-off front over a set of space allocation and timetabling

problems.

In Cowling and Chakhlevitch (2003), a range of hyper-heuristics are studied based

on Simple Random and Greedy heuristic selection methods. According to the descrip-

tion of the Greedy method in Cowling et al (2000), worsening moves are never ac-

cepted. On the other hand, it is possible that all heuristics might worsen the quality of

a candidate solution when Greedy is used. In Cowling and Chakhlevitch (2003), such

situations are allowed. This is a more general approach that allows the move acceptance

component to deal with worsening moves, enriching the generation of different hyper-

heuristics embedding different acceptance mechanisms. In this study, Peckish heuristic

selection strategies are also investigated along with four different Tabu Search based

move acceptance strategies. These strategies accept an improving move and the related

heuristic is removed from the tabu list if it is there. A non-improving move is accepted

only if the employed heuristic is not in the tabu list. The hyper-heuristics utilise Only

Improving, All Moves and a variant of All Moves that discards moves generating the

same objective value as the current solution are utilised as move acceptance criterion.

The approaches were evaluated on a real-world personnel scheduling problem with

ninety five low level heuristics yielding promising results. However, a disadvantage is

that the process for selecting a low-level heuristic to apply at each decision point is

slow since it involves examining all heuristics from a large set. Therefore, in Chakhle-

vitch and Cowling (2005), two learning strategies were investigated for choosing the

subset of the fittest low-level heuristics. At each step, the changes in the quality of a

solution is compiled to reflect the total improvement due to a heuristic. Greedy–Tabu

Search (event based tabu list) that linearly reduces the number of the fittest low level

heuristics turned out to be the most promising hyper-heuristic.

4.2.3 Hyper-heuristics using heuristic selection with no learning and

non-deterministic move acceptance

In Ayob and Kendall (2003), a Monte Carlo move acceptance strategy is proposed. This

approach accepts all improving moves, while the non-improving moves are accepted

with a certain probability that decreases with respect to the worsening of the objective

value δ = f(snew)− f(sold), assuming a minimisation problem. The authors explored

a Linear (LMC), an Exponential probability function (EMC), and included their most

sophisticated formulation based on the computation time and a counter of consecutive

23

non-improvement iterations (EMCQ):

e−∆fm

Q , (5)

where m is the unit time in minutes and Q is a counter for successive worsening moves.

The EMCQ formulation is similar to that of a simulated annealing approach (Kirk-

patrick et al, 1983; Cerny, 1985). The difference, is that it does not include a temper-

ature parameter and thus a cooling schedule. Hyper-heuristics combining Simple Ran-

dom and {Linear, Exponential, EMCQ} are applied to optimise the scheduling of elec-

tronic component placement on a printed circuit board. Their performances are com-

pared to the combination of {Simple Random, Choice Function} and {All Move, Only

Improving} hyper-heuristics. Simple Random–EMCQ having a non-deterministic ac-

ceptance strategy delivered a superior performance as compared to the hyper-heuristics

using deterministic acceptance with and without learning for the given problem in-

stances. Although, it seems no parameter tuning is necessary, more instructions will

be executed in a unit time on a faster machine as compared to a slower machine.

Consequently, different results might be produced for the same number of iterations.

In Kendall and Mohamad (2004a), a variant of the Great Deluge acceptance criteria

Dueck (1993) is incorporated within a hyper-heuristic framework. In this acceptance

strategy, at each iteration, any configuration is accepted which is not much worse

than an expected objective value, referred to as level, which changes at a linear rate

every step. If f(snew) < level = f(s0) − (t∆F )/T , then the move is accepted at

step t, where f(s0) is the objective value of the initial solution, ∆F is the difference

between the objective values of f(s0) and the expected final objective value and T is

the maximum number of iterations. Heuristic selection is performed randomly. Simple

Random–Great Deluge generated competitive results as compared to a constructive

heuristic and a genetic algorithm for solving channel assignment benchmark problems,

a real world-problem from the mobile communications industry.

In another study, Kendall and Mohamad (2004b) extends the Record-to-Record

Travel acceptance criteria of Dueck (1993) suitably modified to be used as a hyper-

heuristic. Any new candidate solution is accepted which is not much worse than the

current one within a given fixed limit, referred to as deviation. If δ = f(snew) −f(sold) < deviation (assuming a minimisation problem), the algorithm will accept

the new solution. If an accepted move is an improving one, then the quality of the

solution is updated as well. Simple Random–Record-to-Record Travel hyper-heuristic

is also applied to benchmark instances of a channel assignment problem. The empirical

results suggest that this hyper-heuristic is superior to using All Move, Only Imroving

and EMCQ move acceptance strategies. The approach is also compared to two problem

specific state-of-the-art algorithms for the channel assignment problem: a constructive

heuristic and a genetic algorithm. The performance of Simple Random–Record-to-

Record Travel turns out to be comparable to the previous studies.

A Simulated Annealing acceptance method in hyper-heuristics is investigated in

Bai and Kendall (2005). The approach is studied on a shelf space allocation problem.

In Simulated Annealing, the improving solutions are always accepted, and worsening

moves are accepted according to the Metropolis criterion, defined by exp(−δ/τ) where

δ = f(snew) − f(sold) is the change in the objective function, and τ is the tempera-

ture parameter. The temperature is decreased during the algorithm run using a cooling

schedule. The authors discuss how to compute the relevant Simulated Annealing param-

eters automatically. Different approaches are allowed to improve an initial candidate

24

solution that is generated by a greedy heuristic. The results show that the Simple

Random–Simulated Annealing hyper-heuristics outperform Simple Random–Only Im-

proving, Simple Random–All Moves, Greedy–Only Improving, Choice Function–All

Moves, two conventional simulated annealing approaches each using a different single

neighbourhood operator in all problem instances tested. Two strategies to decide the

initial temperature are compared. One of them computes the initial temperature as

a factor of the initial objective value, while the other one samples a set of random

solutions and makes the computation based on the largest objective difference. The

former scheme performs slightly better than the latter one.

Antunes et al (2009) describes a multi-objective Simple Random–Simulated An-

nealing hyper-heuristic for solving a power compensation problem in electricity distri-

bution networks. Deciding the location of network nodes and the size of capacitors to

be installed for reactive power compensation requires two conflicting objectives to be

achieved; namely, cost and power loss. Six low level heuristics are designed to make a

move from one feasible solution to another. Simulated Annealing makes its acceptance

decision based on the dominance between the new solution and the archived solutions.

The weighted sum of two objective values is used in the acceptance probability function

whenever needed. The results indicate that this hyper-heuristic performs slightly worse

than a multi-objective genetic algorithm.

4.2.4 Hyper-heuristics using heuristic selection with on-line learning and

non-deterministic move acceptance

In Dowsland et al (2007), a variant of Reinforcement Learning with Tabu Search

(RLTS) is hybridised with a Simulated Annealing with Reheating move acceptance

strategy. In particular, RLTS is modified in such a way that a single application of a

low-level heuristic h, is defined to be k iterations of h. Therefore, the decision points

are set every k iterations, and the feedback for updating the performance of heuristic h

is based on the best cost obtained during those k iterations. Additionally, an undulat-

ing cooling schedule is proposed as a mean to deal with the effects of having different

neighbourhood sizes (given by the pool of low-level heuristics used). The temperature

is reduced based on the schedule in Lundy and Mees (1986):

t =t

1 + βt, (6)

A reheating scheme is employed after a rejected move and the required acceptance rate

r is reduced periodically as discussed in Thompson and Dowsland (1996) at every k iter-

ations. In a way, the updates of scores for low level heuristics in Reinforcement Learning

with Tabu Search and reductions of the acceptance rate in Simulated Annealing with

Reheating are performed together. The proposed hyper-heuristic is applied to a pack-

ing problem of determining shipper sizes for storage and transportation. Real-world

data from a cosmetics company is used as a base for generating experimental data.

The Reinforcement Learning with Tabu Search–Simulated Annealing with Reheating

is superior in performance than a simpler local search strategy (random descent).

Bai et al (2007a) presents a different hyper-heuristic scheme that is possibly inspired

from the studies provided in Burke et al (2003b); Bai and Kendall (2005); Dowsland

et al (2007). The proposed hyper-heuristic uses a reinforcement learning mechanism

with a short term memory as a heuristic selection component. Each heuristic receives

a weight (score) that is updated periodically. A heuristic is selected based on a roulette

25

wheel strategy after these weights are converted into probabilities as described above.

In this study, a different Simulated Annealing with Reheating scheme is used as a

move acceptance method which executes switching between annealing and reheating

phases during the search. During the annealing phase, e−∆f

t is used as part of the

acceptance function and the temperature is reduced using the Equation 6, where β =(t0−tfinal)itemp

maxIterations·t0·tfinaland itemp is the number of iterations at a temperature. During

the reheating phase, t = t1−βt is used to increase the temperature to a value when the

last improvement has been observed. Then the system enters an annealing phase again.

During this phases, the percentage of accepted calls is used as the weight of a given low

level heuristic, while the percentage of heuristic invocations yielding new solutions is

used as its weight during the reheating phases. The proposed hyper-heuristic is tested

on nurse rostering, course timetabling and bin packing problems and comparisons to

previously proposed approaches show that it is competitive.

In Pisinger and Ropke (2007), a competent unified methodology is presented for

solving different vehicle routing problems. The proposed approach extends the large

neighbourhood search framework presented in Shaw (1998) with an adaptive layer.

This layer adaptively chooses among a number of insertion and removal heuristics to

intensify and diversify the search, according to scores for each heuristic accumulated

during the iterations. The hyper-heuristic combines the adaptive heuristic selection

mechanism with a standard Simulated Annealing acceptance strategy based on a linear

cooling rate. A heuristic causing a rejected move gets punished with no increase in its

recent score, while an accepted move is rewarded. The rate of reward for an accepted

move is higher if the resulting solution is a new one. A large number of tests were

performed on standard benchmarks from the literature covering five variants of the

vehicle routing problem. The results proved highly promising, as the methodology was

able to improve on the best known solutions on some instances.

Bhanu and Gopalan (2008) combine a genetic algorithm, and a set of its hy-

brids with simulated annealing, tabu search and hill climbing, respectively under a

Greedy–Great Deluge hyper-heuristic to schedule jobs in a grid environment. The

hyper-heuristic performs better than each individual low level metaheuristic over a

small set of problems.

Ozcan et al (2009) investigates different heuristic selection methods that will per-

form the best in combination with a recently proposed acceptance criteria, namely;

Late Acceptance strategy (Burke and Bykov, 2008). This method is a memory based

technique that maintains the history of objective values from the previous solutions

in a list of size L. The new solution is compared to a previous solution obtained at

the Lth step and the acceptance decision is made accordingly. The results show that

Simple Random performs the best when combined with Late Acceptance as compared

to Greedy, Choice Function, Reinforcement Learning and Reinforcement Learning with

Tabu Search. The delay within the acceptance strategy seems to deceive the learning

mechanisms. Greedy turns out to be the worst heuristic selection choice to be combined

with Late Acceptance.

4.2.5 Comparison studies

In Bilgin et al (2006), an empirical study is carried out and thirty five hyper-heuristics

resulting from the combinations of previously proposed heuristic selection and accep-

tance strategies are compared: {Simple Random, Random Gradient, Random Permuta-

26

tion, Random Permutation Gradient, Greedy, Choice Function, Reinforcement Learn-

ing with Tabu Search} versus {All Moves, Only Improving, Improving and Equal, Great

Deluge, EMCQ}. The hyper-heuristics are evaluated on a set of function optimisation,

and timetabling benchmark instances. On function optimisation benchmarks, none of

the combinations of heuristic selection and acceptance strategy is found to dominate

over the others. However, Improving and Equal produces better average performance.

It is observed that the performance differences between heuristic selection methods are

not as significant as it is between the acceptance criteria. The Choice Function heuris-

tic selection scheme produced a slightly better average performance. On the exam

timetabling instances, the Choice Function–EMCQ and Simple Random–Great Deluge

produce better quality solutions than the rest of the hyper-heuristic combinations.

Ozcan et al (2006) investigates the performance of different hyper-heuristic frame-

works. The low level heuristics are classified as either a hill climber (meme) that aims

to generate an improved solution or a mutational heuristic that perturbs a candidate

solution without considering whether the resulting solution will be improved or not.

Three alternative hyper-heuristic frameworks to the standard framework, which will

be referred to as type FA, are proposed that allows inclusion and/or separation of the

hill-climbing process. In the standard framework, a hyper-heuristic manages a single

set of low level heuristics which might include mutational and hill climbing heuristics

mutually. More details on the standard framework is provided in Section 4.2. The FB

framework extends FA with an additional hill climbing stage. During the heuristic se-

lection process in FB either a hill climber or a mutational heuristic might be chosen.

If a hill climber is selected, the framework continues with the acceptance as in FA. If a

mutational heuristic is selected, then a prefixed (single) hill-climber is applied to the so-

lution. This framework guarantees that a mutated solution goes through a hill climbing

process using possibly the best heuristic that works well with the mutational heuristics

defined in the system. In the FC framework, the low level heuristic set contains only

mutational heuristics and a predefined hill climber is applied after each invocation of

the selected heuristic. The FD framework is the most general framework that uses

two hyper-heuristics successively for managing mutational heuristics and hill climbers,

respectively. The experimental results over benchmark functions suggest that using a

different hyper-heuristic framework in case of single or multiple hill climbers might

have a positive impact on the hyper-heuristic performance. Considering the average

performance of the frameworks, FC turns out to be the most promising one.

Ozcan et al (2008) extend the previous studies of (Bilgin et al, 2006; Ozcan et al,

2006). Based on the observation that some low level heuristics are rarely chosen if a

learning heuristic selection scheme is used, the benchmark function optimisation ex-

periments are repeated using a reduced set of mutational heuristics and hill climbers.

The results show that the choice of single hill climber affects the performances of the

FB and the FC frameworks. With the best hill climber, both frameworks perform

significantly better than the others. FC with the best hill climber delivers a much bet-

ter performance as compared to genetic algorithm. Moreover, its performance turns

out to be comparable to memetic algorithms. A memetic algorithm embedding an

Simple Random–Improving and Equal hyper-heuristic which is categorised as a static

external-level adaptation mechanism in Ong et al (2006a), also performs well. More in-

vestigations on memetic algorithms embedding hyper-heuristics to choose hill climbers

can be found in Ersoy et al (2007).

Bai et al (2008) investigates the performance of a set of fast approaches for solving

a fresh produce inventory and shelf space allocation problem and compares against a

27

multi-start generalised reduced gradient algorithm proposed in Bai and Kendall (2008).

Four different greedy heuristics, GRASP (Feo and Resende, 1995), traditional simulated

annealing using a single neighbourhood operator, Reinforcement Learning with Tabu

Search–All Moves (Burke et al, 2003b), Simple Random–Simulated Annealing (Bai and

Kendall, 2005) and Reinforcement Learning with Tabu Search–Simulated Annealing

that combines components from the former two hyper-heuristics are implemented. A

comparison between hyper-heuristics shows that Simulated Annealing based hyper-

heuristics performs better than Reinforcement Learning with Tabu Search–All Moves.

Moreover, they deliver a similar performance to the traditional simulated annealing

approach and GRASP. As a heuristic selection component, Reinforcement Learning

with Tabu Search generates a slightly better performance than Simple Random within

the hyper-heuristic framework.

In their study, Bai et al (2007a) synchronise parameter updates for heuristic se-

lection and move acceptance methods, as if both hyper-heuristic components were

indivisible and required to operate in that manner. In fact, they are separable. Burke

et al (2008) compares the performances of hyper-heuristics using a Monte Carlo move

acceptance criteria. In this study, combinations of {Simple Random, Choice Function}versus {Simulated Annealing, Simulated Annealing with Reheating, EMCQ}, Greedy–

Simulated Annealing with Reheating and the proposed hyper-heuristic in Bai et al

(2007a) are tested on examination timetabling problems yielding the success of Choice

Function–Simulated Annealing with Reheating. Nareyek (2003) shows that using max

strategy for heuristic selection performs better. The Choice Function makes use of this

strategy, while the learning mechanism in Bai et al (2007a) does not. Moreover, this

mechanism rewards a heuristic based on its acceptance ignoring whether an improve-

ment is achieved or not. This strategy combined with a short term memory might be

misleading the heuristic selection process.

4.2.6 Multi-point search based hyper-heuristics

In Cowling et al (2002a), an indirect genetic algorithm for solving a personnel schedul-

ing problem is proposed. The approach can be regarded as a hyper-heuristic that uses

a GA as the heuristic selection mechanism. The genetic representation is given by a se-

quence of integers each of which represents a single low-level heuristic. Each individual

in the population gives a sequence of heuristic choices that indicates which low level

heuristic to apply, and in what order. Extensions to the original approach are proposed

that consider adaptive length chromosomes, and guided operators (Han and Kendall,

2003). This approach produces better results on the trainer scheduling problem when

compared to both a direct encoding genetic algorithm and a memetic algorithm. It also

outperforms its component heuristics.

The ant colony algorithm is used as a hyper-heuristic in Burke et al (2005b) and

Chen et al (2007), to solve a personnel scheduling and a sports timetabling problem,

respectively, with promising results. Each vertex in the ant algorithm represents a

low-level heuristic. A number of ants (or hyper-heuristic agents) are located uniformly

among the vertices and carry initial solutions. In their path through the network, the

ants apply the low-level heuristic at each node. The application to the sport timetabling

problem (Chen et al, 2007), is able to produce good quality solutions for the studied

instances, even outperforming best-known results on some of the instances.

Crainic and Toulouse (2003) classifies cooperative search methods as type 3 parallel

strategies which allow multiple search methodologies to guide the search process via

28

information sharing in a multi-thread environment. These strategies can be thought

of as parallel hyper-heuristics which have been increasingly used to combine multiple

low level (meta-)heuristics. Ouelhadj and Petrovic (2008) and Ouelhadj and Petrovic

(2009) propose an agent based hyper-heuristic framework in which the information

sharing process is handled synchronously and asynchronously, to utilise multiple low

level heuristics for solving a permutation flow shop problem. Ouelhadj and Petrovic

(2010) extends these studies further. A set of low level heuristic agents process multiple

candidate solutions in parallel in both of these approaches. A list of elite solutions is

built and maintained by the low level heuristics under a coordinator hyper-heuristic

agent. The results show that the cooperative method based on asynchronous com-

munication between the heuristic agents and the coordinator hyper-heuristic agent

outperforms the synchronised communication version.

Meignan et al (2010) describes another self-adaptive agent based approach, referred

to as Coalition-Based Metaheuristic. Agents are grouped into coalitions to explore the

search space of solutions concurrently. Each agent modifies a solution using a set of low

level heuristics. The hyper-heuristic level within the proposed framework is associated

with the Guide and Strategist roles which interact. Here the Guide role schedules, selects

or composes low-level components. Strategist role corresponds to learning mechanisms

that modify the strategy of the Guide role The heuristic selection is performed by

Guide based on a reinforcement learning strategy (Pisinger and Ropke, 2007), which is

managed by Strategist in an online manner. The agents communicate asynchronously.

They are allowed to broadcast a solution if it is improved and share their internal

decision rules for mimetism. Two types of operators are distinguished as diversifiers

(mutational heuristics) and intensifiers (local search heuristics) in this system. The au-

thors investigate the performance of Coalition-Based Metaheuristic over Christpphides

benchamrks for vehicle routing problem. The authors use some construction heuristics

to generate initial solutions, crossover operators and mutation as diversifier low level

heuristics. Additionally, six gradient descent hill climbers are used based on different

types of neighbourhoods, such as 2-opt. The results show that the use of reinforcement

learning and mimetism improves the quality of solutions and Coalition-Based Meta-

heuristic delivers a superior performance over not all, but some previously proposed

approaches.

Biazzini et al (2009) provide a comparison of distributed hyper-heuristic strategies

based on an island model in which independent processors runs the same algorithm.

The low level (meta-)heuristics contain six variants of Differential Evolution, Particle

Swarm Optimisation and random sampling for real parameter optimisation. The test

results indicate that distributed hyper-heuristics perform well over a set of benchmark

functions and two realistic space-probe trajectory optimisation problems.

4.2.7 Summary and discussion

Most of the existing hyper-heuristics to select perturbative heuristics are designed

based on a single point search framework. Initial studies concentrate on utilising dif-

ferent mechanisms to manage a set of low level heuristics. As hyper-heuristics were

initially defined as “heuristics to choose heuristics”, almost no emphasis is given to

the acceptance mechanisms during these initial studies. Later, it has been observed

that by using more sophisticated move acceptance criteria, the performance can be

improved substantially. After the initial studies, there has been a rapid growth in the

usage of well known acceptance methods in different hyper-heuristic frameworks. Most

29

frequently used and successful heuristic selection methods are Choice Function (Cowl-

ing et al, 2000, 2002b) and reinforcement learning variants (Nareyek, 2003; Burke et al,

2003b; Dowsland et al, 2007). Simulated annealing (Bai and Kendall, 2005; Bilgin et al,

2006; Dowsland et al, 2007; Bai et al, 2007a) and great deluge variants (Kendall and

Mohamad, 2004a; Bilgin et al, 2006) turns out to be the best choices as move accep-

tance components to be used within hyper-heuristics to select perturbative heuristics.

It appears that the choice of move acceptance component is slightly more important

than the choice of heuristic selection. One of the reasons for this might be the total

number of low level heuristics used within hyper-heuristics apart from their nature. In

most of the applications of hyper-heuristics to different problem domains, the number

of low level heuristics does not exceed a factor of ten. In fact, Cowling and Chakhle-

vitch (2003); Chakhlevitch and Cowling (2005) are the only studies that employs a

large set of low level perturbative heuristics within a heuristic selection hyper-heuristic

framework. In order to observe how well the proposed hyper-heuristics generalise, they

need to be applied to other problem domains. We can expect more comparative studies

in the future. One of the goals of hyper-heuristic research is raising the level of gen-

erality. As a general problem solver, a hyper-heuristic does not aim to outperform a

custom made solver for a given problem. Its applicability over a wide range of prob-

lem domains is more crucial. For this reason, performance evaluation measures are of

interest, particularly in empirical comparison studies.

In order to generate better learning schemes that will improve the decision making

process during heuristic selection, one should always remember that a learning process

involves memory. Memory length can be handled in a different way. For example, the

scores for low level heuristics are updated based on the entire historical information

in Burke et al (2003b); Dowsland et al (2007). On the other hand, the minimum

and maximum scores are bounded in Nareyek (2003). This approach serves as some

type of a forgetting mechanism for successive improving or non-improving moves. The

empirical study carried out in Bai et al (2007b) on university course timetabling, shows

that hyper-heuristics using a short-term memory produces better results than both an

algorithm without memory and an algorithm with infinite memory. The memory length

is found to be sensitive to different problem instances.

Combining simple, yet effective problem dependent low level perturbative heuristics

with intelligent heuristic selection and move acceptance methods in a single framework

provides a powerful search and optimisation tool. Hyper-heuristics can be used for

solving any type of complex real world problems. They are easy to implement. Since,

the main components of a hyper-heuristic process only problem domain independent

information to perform search, they require no modification whatsoever to be used in

a different problem domain. The studies on parallel and distributed processing strate-

gies can benefit from the hyper-heuristic research and vice versa. These methodologies

can handle single objective as well as multi-objective problems. The empirical inves-

tigations up to now shows that hyper-heuristics to select perturbative heuristics are

fast techniques that produce solutions with reasonable quality in a reasonable time.

Moreover, their performance is comparable to the bespoke systems.

5 Modern heuristic generation methodologies

The previous section covered heuristic selection methodologies. In contrast, this section

will review another class of hyper-heuristics: heuristic generation methodologies. The

30

defining feature of this class is that the hyper-heuristic searches a space of heuristics

constructed from components, rather than a space of complete, pre-defined, heuristics

(see section 3). While both classes output a solution at the end of a run, a heuristic

generator also outputs the new heuristic that produced the solution, and this heuristic

can be potentially reused on new problem instances.

Genetic programming (Koza, 1992; Koza and Poli, 2005) is an evolutionary com-

putation technique that evolves a population of computer programs, and is the most

common methodology used in the literature to automatically generate heuristics. In the

case that the evolved programs are heuristics, genetic programming can be viewed as a

hyper-heuristic to generate heuristics. However, genetic programming is not inherently

a hyper-heuristic, as the evolved programs can also directly represent problem solu-

tions. For example, in symbolic regression, the solution is a formula, and the ‘programs’

in the population are candidate formulas, which are not used as heuristics. As another

example, genetic programming can be employed to evolve programs which construct

the design of artifacts such as bridges, circuits, and lenses. These programs are not

heuristics, but a series of deterministic instructions.

Automatically generated heuristics may be ‘disposable’ in the sense that they are

created for just one problem, and are not intended for use on unseen problems. This

terminology was first used by Bader-El-Din and Poli (2007). Alternatively, the heuristic

may be created for the purpose of reusing it on new unseen problems of a certain class.

It is generally the case that all heuristics generated by a hyper-heuristic can technically

be defined as reusable, as they can be applied to a new instance to produce a legal

solution. However, they may not perform well on new instances if the particular hyper-

heuristic methodology has not been designed with reusability in mind. For a generated

heuristic to be successful when reused, the hyper-heuristic would usually train it offline,

on a set of representative problem instances.

There are a number of potential advantages of automatically generating heuristics.

The characteristics of problem instances vary, and obtaining the best possible result

for an instance would ideally require a new heuristic specialised to that instance, or

a specialised variation of a previously created heuristic. It is inefficient for a human

analyst to specialise heuristics on a per-instance basis. As such, human created heuris-

tics are rarely successful on only one problem instance, they are usually designed to

be effective on a given class of problems. In contrast, an automated heuristic design

process makes it potentially feasible, and cost effective, to design a heuristic for each

problem instance. As the process is automated, it is less demanding on human resources

and time. As it is more specialised, a generated heuristic could even produce a better

solution than that which can be obtained by any current human created heuristic, and

many such examples are discussed in this section.

For example, ‘best-fit’ is a human created heuristic for one dimensional bin packing,

which performs well on a wide range of bin packing instances. It was created as a general

heuristic for all bin packing problems, and no heuristic is superior in both the average

and worst case (Kenyon, 1996). However, over a narrower set of bin packing problems

with piece sizes defined over a certain distribution, best-fit can be outperformed by

automatically generated heuristics which are ‘tailored’ to the distribution of piece sizes

(Burke et al, 2007a).

Genetic programming is not the only methodology used as a hyper-heuristic to

generate heuristics. Squeaky wheel optimisation (Joslin and Clements, 1999; Aickelin

et al, 2009) can be considered as a hyper-heuristic which iteratively refines a heuristic

for the instance it is solving. When solving a scheduling problem, there are a number

31

Application Domain References

Production Scheduling

Jakobovic et al (2007)

Ho and Tay (2005)

Tay and Ho (2008)

Dimopoulos and Zalzala (2001)

Geiger et al (2006)

Cutting and Packing

Burke et al (2006a, 2007a,b)

Poli et al (2007)

Kumar et al (2008)

Allen et al (2009)

Satisfiability

Fukunaga (2002, 2004, 2008)

Bader-El-Din and Poli (2007, 2008)

Lokketangen and Olsson (2010)

Travelling Salesman Problem

Keller and Poli (2007b,a, 2008b,c,a)

Oltean and Dumitrescu (2004)

Runka (2009)

Function Optimisation

Oltean (2005)

Oltean and Grosan (2003)

Tavares et al (2004)

Timetabling and Scheduling

Pillay (2009)

Bader-El-Din and Poli (2009)

Aickelin et al (2009)

Additional Domains

Drechsler and Becker (1995)

Drechsler et al (1996)

Minton (1996)

Schmiedle et al (2002)

Stephenson et al (2003)

Oltean (2005)

DiGaspero and Schaerf (2007)

Kumar et al (2009)

Table 5 Application domains of heuristic generation methodologies.

of well known heuristics which order the events with some metric, and then construct

the solution using that ordering. A squeaky wheel optimisation approach performs this

process iteratively, and analyses the solution at each iteration. The results of this anal-

ysis are utilised to adaptively order the events for the next iteration. The methodology

performs a search over the space of event orderings, each of which represents a different

heuristic.

Table 5 presents a summary of the papers which involve the automatic genera-

tion of heuristics. The rest of the section is organised by application area as follows;

production scheduling (section 5.1), cutting and packing (section 5.2), SAT (section

5.3), the travelling salesman problem (section 5.4), function optimisation (section 5.5),

timetabling and scheduling (section 5.6). These are the domains most widely studied

in the literature. Heuristics evolved for other domains are discussed in section 5.7, and

conclusions are given in section 5.8.

32

5.1 Production scheduling

Genetic programming has rarely been used to solve production scheduling instances

directly, because it is unsuitable for encoding solutions. However, it is highly suitable

for encoding scheduling heuristics (Jakobovic et al, 2007). The evolution of dispatching

rules is the most common application of hyper-heuristics in this domain.

Ho and Tay (2005) and Tay and Ho (2008) employ genetic programming to evolve

composite dispatching rules for the flexible job shop scheduling problem. Jakobovic

et al (2007) employ the same technique for the parallel machine scheduling problem.

The evolved dispatching rules are functions, which assign a score to a job based on the

state of the problem. When a machine becomes idle, the dispatching rule is evaluated

once for each job in the machine’s queue, and each result is assigned to the job as its

score. The job in the queue with the highest score is the next job to be assigned to the

machine.

The terminals of the genetic programming algorithm are components of previously

published dispatching rules, and the functions are the arithmetic operators and an

automatically defined function (see Koza, 1994; Koza and Poli, 2005). The results ob-

tained by the best evolved dispatching rule are better on over 85% of the instances. This

shows that genetic programming can combine and rearrange heuristic components, to

create heuristics superior to those which have been created by humans. Importantly, the

heuristics are shown to be reusable on new problem instances because an appropriate

training set is used to train them during their evolution.

Dimopoulos and Zalzala (2001) evolve priority dispatching rules for the single ma-

chine scheduling problem, to minimise the total tardiness of jobs. The terminal set is

based on the human designed ‘Montagne’ dispatching rule, and contains five elements,

representing both global and local job information. The function set consists of the four

basic arithmetic operators. While the function and terminal sets are relatively simple,

the system evolves heuristics superior to the Montagne, ADD, and SPT heuristics.

Geiger et al (2006) also employ genetic programming to evolve dispatching rules for

single machine problems. The function and terminal sets are expanded from that pre-

sented by Ho and Tay (2005) and Dimopoulos and Zalzala (2001). Human competitive

heuristics are produced under a variety of scheduling conditions, often replicating the

human created heuristics for the problems. The system also obtains human-competitive

results on a two machine flowshop problem, where a unique dispatching rule is evolved

for each machine simultaneously.

5.2 Cutting and packing

Burke et al (2006a, 2007a,b) employ a genetic programming hyper-heuristic method-

ology, to generate heuristics for one dimensional bin packing. The heuristics generated

by this system are functions consisting of arithmetic operators and properties of the

pieces and bins. The heuristics operate within a fixed framework that packs the pieces

of an instance one at a time. For each piece in turn, the framework iterates through the

bins, applying the heuristic once to each. The heuristic returns a value for each bin. In

initial research, the piece is placed into the first bin which receives a value greater than

zero (Burke et al, 2006a). A modification of the system is later presented, where the

bin which receieves the highest value is the one into which the piece is placed (Burke

et al, 2007a,b).

33

Burke et al find that better results can be obtained by putting the piece into the bin

with the highest score. All of the bins are considered by the heuristic with this method,

and so the heuristic has a greater choice of bins, allowing heuristics to evolve which

exceed the performance of the ‘best-fit’ heuristic. The generated heuristics cannot be

superior to best-fit in general (Kenyon, 1996), but they are shown to be better than

best-fit on new instances of the same type as those they are trained on.

These heuristics maintain their performance on new instances much larger than the

training set in Burke et al (2007b). This work shows that there is a trade-off between

the time taken to evolve a heuristic on larger instances, and the heuristic’s scalability.

Poli et al (2007) also employ genetic programming to evolve heuristics for one-

dimensional bin packing. The structure within which their heuristics operate is based

on matching the piece size histogram to the bin gap histogram, and is motivated by

the observation that space is wasted if, when placing a piece into a bin, the remaining

space is smaller than the size of the smallest piece still to be packed. For example, if the

smallest piece still to be packed is of size 20, and there is a gap of size 19 available, this

gap will never be filled and the space is wasted. In addition to the histogram approach,

the work further differs from that of Burke et al., as linear genetic programming is

employed to evolve the heuristics, and the problem is offline bin packing, rather than

online.

Constructive heuristics for the two dimensional strip packing problem have been

evolved with genetic programming in Burke et al (2010), which are competitive with

the best human created constructive heuristic in the literature. In contrast to their

previous work on one dimensional bin packing, the heuristics operate on the offline

problem, where all of the pieces to be packed are known. The heuristics choose the

most appropriate piece to pack next, and where to place it in the solution.

Allen et al (2009) evolve heuristics for three dimensional knapsack packing. Their

performance is compared to the human created best-fit heuristic and a simulated an-

nealing methodology. While the evolved heuristics are worse than best-fit on most

instances, they are competitive with the simulated annealing method.

Recent work by Kumar et al (2008) presents a genetic programming system which

evolves heuristics for the biobjective knapsack problem. This is the first paper in which

heuristics for a multiobjective problem have been automatically generated with a hyper-

heuristic. To pack a knapsack instance, an evolved heuristic iterates through the list of

pieces still to be packed, and is evaluated on each, using the profit and weight of the

piece as inputs. When an evaluation returns a value of greater than or equal to one,

then the iteration stops and that piece is packed. This is similar to the bin packing

methodology of Burke et al (2006a), as it uses a threshold to make a decision before

all of the options have been evaluated.

5.3 Boolean satisfiability (SAT)

Fukunaga presents ‘CLASS’ (Composite Learned Algorithms for SAT Search), an au-

tomated heuristic discovery system for the SAT problem. Earlier papers by Fukunaga

(2002, 2004) represent the initial work, while much more analysis is given in (Fuku-

naga, 2008). SAT is a domain where the most successful heuristics from the literature

have a similar structure. Indeed, better heuristics have been created simply by adding

a ‘random walk’ element to an existing heuristic. Fukunaga has broken this struc-

ture down into component parts, and the CLASS system is a genetic programming

34

methodology used to evolve human competitive heuristics consisting of these compo-

nents. Amongst others, there are some components which supply a set of variables,

some of which select a variable from such a set, and some which make use of conditions

to decide which subset of components to execute. Fukunaga (2008) shows that certain

human created heuristics from the literature, such as GWSAT and WalkSAT, can be

represented with this component set. Fukunaga states that, because of the number of

possibilities involved, the task of combining the components to create effective new

heuristics is difficult for humans, but well suited for an automated system.

Fukunaga (2002, 2004, 2008) does not employ the genetic programming operators

of crossover and mutation in their standard form. Instead of standard crossover, indi-

viduals are combined with a conditional operator, which keeps the original individuals

intact and ‘blends’ their behaviour. If the condition is met, one individual is executed,

else the other is executed. A form of mutation occurs when the combined trees violate

a given maximum depth constraint.

Bader-El-Din and Poli (2007) observe that this results in heuristics consisting of

other nested heuristics. The heuristics are composites of those in early generations,

and are therefore relatively slow to execute. Bader-El-Din and Poli present a different

heuristic generation methodology for SAT, which makes use of traditional crossover

and mutation operators to produce heuristics which are more parsimonious, and faster

to execute. A grammar is defined, which can express four existing human created

heuristics, and allows significant flexibility to create completely new heuristics.

SAT is also the test domain for a proof of concept of the inc* system, presented by

Bader-El-Din and Poli (2008). The inc* methodology incrementally solves a sequence

of progressively more complex instances with a fixed human created heuristic. An

instance is decomposed into a much simpler instance to solve, before using the result

to solve one slightly more complex. The difficulty of the instance is incrementally

increased in this way until the original instance is reached and solved. The example

given in the paper is to progressively add clauses to SAT instances. The inc* algorithm

involves backtracking in the situation where an instance is judged too difficult, and the

problem is made simpler again. Genetic programming is used to evolve the heuristic

which decides how many clauses to remove from the instance when backtracking, and

how many to add to the instance when making it more complex. The inc* methodology

as a concept is potentially general over many problem domains, but the heuristic to

simplify the problem is necessarily problem specific. The genetic programming operates

as a hyper-heuristic in this case because it searches the space of these problem specific

heuristics.

ADATE is a methodology which generates code in a subset of the functional pro-

gramming language ML. Lokketangen and Olsson (2010) show how this technique can

be utilised to automatically generate metaheuristic code. They apply the methodol-

ogy to the boolean optimisation problem (BOOP), which can be regarded as a SAT

problem with an additional objective, as there is a profit associated with each vari-

able having a true or false value (Lokketangen and Olsson, 2010). They begin with an

ML implementation of a tabu-search metaheuristic from the literature, and ADATE

modifies the section of the code that decides which variable to flip next. This is an

example of a common methodology, to begin with an existing algorithm as a tem-

plate, and allow the automatic code generator to modify the ‘heuristic’ sections of the

algorithm which make the decisions during a search. The majority of the literature em-

ploys genetic algorithms or genetic programming for automatic code generation, but

the ADATE mechanism is different. While ADATE modifies code iteratively with each

35

new individual being created from a ‘parent’, it does not make use of crossover. The

ADATE ‘replacement’ operator is similar to mutation in genetic programming, but it

synthesises new code through a less stochastic process.

5.4 Travelling Salesman Problem

Keller and Poli (2007b) present a linear genetic programming hyper-heuristic for the

travelling salesman problem (TSP). The hyper-heuristic evolves programs which rep-

resent the repeated application of a number of simple local search operations. The

programs are sentences of a language defined by a grammar, which is progressively

made more complex.

The system is first shown to evolve sequences of swap heuristics which perform

better than random search. The two swap heuristics involve rearranging two or three

pairs of edges. Conditional and loop components are then added to the grammar, to

increase the complexity of the evolved heuristics. The results obtained by the evolved

heuristics are within 1% of a state of the art hybrid genetic algorithm, which obtains the

best known results on those two instances. Further analysis of the algorithm is provided

in Keller and Poli (2007a), with regards to the computational resource utilisation.

The loop component in the grammar is controlled by one parameter, which de-

fines how many times the loop will be executed. Keller and Poli (2008b) show the

improvement in efficiency when this parameter is evolved along with the heuristic it-

self. The quality of the results does not improve, but the number of low-level heuristic

calls decreases. It is not shown if the evolved heuristics can be reused successfully on

new problem instances. However, they are successful as disposable heuristics on the

instance they were evolved for, and it is shown that good heuristics are consistently

evolved when the genetic programming hyper-heuristic is run numerous times.

Keller and Poli (2008c,a) further extend their grammar by adding a ‘subheuristic’

component. This is a component which is a constituent part of a heuristic. Specifically,

they add one component to the grammar which swaps the order of two nodes in the

tour. This increases the efficiency of the search, with less heuristic calls needed to

produce the same quality of solution.

Also for the travelling salesman problem, Oltean and Dumitrescu (2004) evolve

constructive heuristics, as opposed to the local search heuristics evolved by Keller

and Poli (2007b). They use multi expression programming to evolve functions that

decide which node of the graph to add to the tour. This is another example of the

common technique of evolving a scoring function which operates within a fixed iterative

framework. The decision-maker is evolved, but the context within which the decision is

made remains fixed. In general, at each decision point, the function is evaluated on all

available options to obtain a score for each. The option with the highest score is the one

that is actually performed on the partial solution. In this case, Oltean and Dumitrescu

(2004) apply the candidate function to all of the cities that are not yet included in the

partial tour, and the one which receives the highest score from the function is added

to the tour.

One general methodology for automatically generating heuristics is to evolve a

fundamental part of an existing metaheuristic search algorithm. This has the potential

to produce a much better variation of the original human designed algorithm. Runka

(2009) presents such a system for evolving the edge selection formula of an ant colony

36

optimisation algorithm. While the evolved formulae were tested on only two unseen

TSP instances, the results were better than the original human designed formula.

5.5 Function Optimisation

Oltean (2005) presents a linear genetic programming hyper-heuristic, which evolves

evolutionary algorithms. The problem domains employed to validate the technique are

function optimisation, the travelling salesman problem, and the quadratic assignment

problem.

A standard individual in a linear genetic program represents a series of instructions

that manipulate values in a memory array. The memory positions are referred to as

‘registers’ A typical instruction could be similar to that shown in equation 7.

reg[1] = reg[0] + reg[7] (7)

Oltean (2005) stores an evolutionary algorithm population in this memory array,

with each member of the population allocated to one register. The genetic operators

are the instructions that operate on the memory array. For example, equations 8 and 9

show two possible instructions which operate on the population array. Each individual

in the genetic programming population therefore represents a method of evolving a

population.

reg[1] = crossover(reg[7], reg[3]) (8)

reg[3] = mutate(reg[5]) (9)

This work is based on earlier work by Oltean and Grosan (2003), which evolves

evolutionary algorithms with multi expression programming. This is similar to linear

genetic programming, but each chromosome can encode multiple evolutionary algo-

rithms as the instructions are not necessarily executed sequentially.

Tavares et al (2004) also present a methodology for evolving an evolutionary algo-

rithm. They specify the main components of a generic evolutionary algorithm, including

initialisation, selection, and the use of genetic operators. Tavares et al explain how each

of these steps, can be evolved individually by a genetic programming hyper-heuristic.

They demonstrate the approach through an example of evolving an effective mapping

function of genotype to phenotype, for a function optimisation problem.

5.6 Timetabling and Scheduling

A grammar based genetic programming system is presented by Bader-El-Din and Poli

(2009) for the exam timetabling problem. The grammar contains elements of graph

colouring heuristics and slot allocation heuristics, and a sentence in this grammar

represents a new heuristic for constructing a timetable. The results returned by the

evolved heuristics are comparable with a range of human created search methodologies

from the literature. The system is presented as an online learning methodology, as it is

not shown whether these evolved heuristics are successful when reused on new problem

instances.

37

The exam timetabling approach of Pillay (2009) evolves a heuristic to order exams.

A sorting algorithm, such as quicksort, has a comparator which decides if one element

should be ahead of another in the list, and then it is applied to as many pairs of

elements as is necessary to create the ordering based on that comparator. Pillay uses

strongly typed genetic programming to evolve the comparator of two elements, which

in this case are exams. The comparator has a tree structure consisting of standard

graph colouring indicators such as largest weighted degree and saturation degree of the

two exams, and logic operators such as ‘less than’ and ‘not equal to’. After sorting, the

exam which appears at the head of the queue is scheduled next.

Squeaky wheel optimisation is applied to personnel scheduling in Aickelin et al

(2009). An ordering of problem components is given to a greedy constructive heuristic to

produce an initial solution. This solution is then analysed to determine the components

which cause problems during the scheduling. The order of components is modified using

this information, and a solution is constructed with this new order. The process iterates

until the stopping condition. One of the goals of hyper-heuristic research is to raise the

level of generality of search methodologies, and this approach is shown to obtain good

results over bus and rail driver scheduling, and nurse scheduling.

5.7 Additional Problem Domains

Schmiedle et al (2002) evolve heuristics for the minimisation of binary decision di-

agrams. A heuristic in this domain can be represented as an ordering of predefined

optimisation modules. Their genetic programming hyper-heuristic evolves an ordering

by assigning the optimisation modules to be terminals. The genetic programming func-

tions are iterative and conditional nodes, which therefore apply the terminal nodes in

a given order.

Stephenson et al (2003) employ genetic programming as a hyper-heuristic to evolve

improved compiler priority functions for three aspects of compiler optimisation. While

they do not use the same terminology, Stephenson et al (2003) perform experiments

evolving disposable and reusable heuristics, and find that extremely good heuristics can

be automatically generated on a per-instance basis. This shows the flexibility of a hyper-

heuristic approach to generating heuristics. Because the output is both a heuristic and a

solution to the training instances, the methodology can simply be an effective method to

generate a solution to just one instance, regardless of reusability. The evolved heuristics

in this study consistently outperform the benchmark human created heuristic in each

of the three compiler optimisation areas.

DiGaspero and Schaerf (2007) present ‘EASYSYN++’, a tool for the automatic

synthesis of stochastic local search algorithm code. The human programmer inputs an

XML description of the algorithm building blocks, and the output is semi complete

C++ classes, which must be completed manually with problem specific code. As such,

the system cannot replace human programming experience (nor should we realistically

expect it to given the current level of technology), and it costs a lot of computational

power, but this is often cheaper than the cost of a human programmer, and so even

partly automated program synthesis can be beneficial. This system differs from the

majority of the literature on automatic heuristic generation, as it outputs actual code

that can be compiled and reused again in a different environment.

Multiobjective genetic programming is used to evolve heuristics for the bounded

diameter minimum spanning tree problem in Kumar et al (2009), focussing on the

38

diameter and cost of the trees as objectives. The evolved heuristics, not the genetic

programming algorithm, generate the pareto set of solutions, and their results are

competitive with, and sometimes superior to, the results of existing heuristics.

5.8 Summary and discussion

This section presented a summary of the literature on heuristic generation method-

ologies. Very promising results have been obtained on a wide variety of optimisation

problems, which have relied on human-generated heuristics thus far. The literature

shows that, typically, evolutionary computation methods are employed to automati-

cally generate heuristics, which are reusable on new problem instances.

A heuristic generation process is often computationally expensive when compared

with a methodology that operates directly on the solution space. However, this is only

a disadvantage in the short term, when results will not be required for future problems.

Consider the application of an evolutionary algorithm directly to the problem space.

The output is just the solution to the instance, and the entire evolutionary algorithm

must be run again if a solution is required for future problems. If the evolutionary

process is employed instead as a hyper-heuristic, to generate a quick reusable heuristic,

then only one run of the evolution is required. The evolved heuristic can then obtain a

comparable result on the future problems, much more quickly than the application of

an evolutionary algorithm. This is one of the main benefits of searching for a solution

method rather than just searching for a solution.

While the evolution process is computationally expensive, it is often quicker than

manual heuristic generation. For example, Geiger et al (2006) state that production

scheduling heuristics from the literature are the result of years of scheduling research,

and the identical evolved heuristic rules are generated within a fraction of this time.

This illustrates one of the main motivations for automatically generating heuristics.

However, the potential components of the evolved heuristics must still be defined

by humans, and it can be argued that successful sets of components are inspired by

the literature on human created heuristics. The research in this new area of automatic

heuristic generation, shows that it is not yet able to completely replace human ingenu-

ity. As Fukunaga (2008) states, humans are able to invent good building blocks, and the

literature does show that hyper-heuristic methodologies have been able to successfully

combine these human-defined building blocks in superior ways.

6 Related Areas

Heuristic search is widely studied in operations research, computer science and artificial

intelligence. A promising direction for developing improved search techniques is to

integrate learning components that can adaptively guide the search. Many techniques

have independently arisen in recent years that exploit either some form of learning, or

search on a configuration space, to improve problem-solving and decision making. We

briefly overview some of these approaches below, categorising them between offline and

online approaches.

39

6.1 Offline approaches

Algorithm configuration: Is concerned with the determination of appropriate val-

ues for free algorithm parameters. It is commonly treated as an optimisation prob-

lem (and therefore as a search problem in a configuration space), where the objec-

tive function captures performance on a fixed set of benchmark instances (Hutter

et al, 2007). Depending on the number and type of parameters, the methods used

to solve this optimisation problem include exhaustive enumeration, beam search

(Minton, 1996), experimental design (Coy et al, 2001; Ridge and Kudenko, 2007),

the application of racing algorithms (Birattari et al, 2002; Birattari, 2004), combina-

tions of fractional experimental design and local search (Adenso-Diaz and Laguna,

2006), and iterated local search (Hutter et al, 2007, 2009).

Meta-learning: The concept of meta-learning (learning about learning) was inspired

by the algorithmic selection problem as formulated by Rice (1976), stated as: Which

algorithm is likely to perform best for my problem? Recognising this problem as a

learning task, the machine learning community developed meta-learning approaches

mainly to learning about classification. The generalisation of meta-learning con-

cepts to constraint satisfaction and optimisation as discussed in (Smith-Miles,

2008b), is closely related to hyper-heuristic research. Examples of these general-

isations can be found in artificial intelligence (Horvitz et al, 2001; Leyton-Brown

et al, 2002) and operations research (Smith-Miles, 2008a).

6.2 Online approaches

Parameter control in evolutionary algorithms: A different approach to algorithm

configuration, also called parameter control, is the idea of tuning algorithm param-

eters on-line at execution time. These approaches were already discussed in section

2. In (Eiben et al, 1999), a useful and widely accepted classification of mechanisms

for parameter control is proposed, together with a detailed literature survey of

work to date in this topic within evolutionary algorithms. More recent surveys and

overviews of the state-of-the- art can be found in (Eiben et al, 2007; Lobo et al,

2007).

Adaptive memetic algorithms: Another approach closely related to heuristic se-

lection based on perturbative heuristics is that of adaptive memetic algorithms, a

recent breed of hybrid evolutionary algorithms, in which several memes (or local

searchers) are available and adaptively selected (or generated altogether) during

the search (Hart et al, 2004; Jakob, 2006, 2002; Krasnogor and Smith, 2000, 2001;

Krasnogor and Gustafson, 2004; Ong and Keane, 2004; Ong et al, 2006a; Smith,

2002, 2007; Neri et al, 2007b,a). In Ong et al (2006b) the authors present a use-

ful classification of memes adaptation in memetic algorithms based on the widely

accepted terminology proposed in (Eiben et al, 1999). This classification consid-

ers the adaptation mechanisms (static, adaptive and self − adaptive) used in the

choice of memes , and the level of historical knowledge employed in the adapta-

tion process (global and local). In self-adaptive meme adaptation, the memes are

encoded as part of the organisms and undergo standard evolution. As discussed in

(Smith, 2008) self- adaptation ideas have been applied to memetic algorithms in

two ways: (i) for self-adapting the choice of local search operators (Krasnogor and

Smith, 2000, 2001; Krasnogor et al, 2002), and (ii) for self-adapting the definition

40

of local search algorithms (Krasnogor and Gustafson, 2002, 2004; Krasnogor, 2004;

Smith, 2002, 2003, 2007). This distinction is analogous to our main classification

of hyper-heuristics into heuristic selection and heuristic generation methodologies

(see section 3).

The main distinction between the hyper-heuristic approaches discussed in section

4.2 and adaptive MAs is that the former conduct a single point search within solu-

tion space, while the latest maintain a population of solutions. Another distinction

is that the low-level heuristics (memes) in memetic algorithms are generally local

searchers, whereas in hyper-heuristics, simple move operators have been primarily

considered. However, there are strong connections between these two classes of al-

gorithms as the challenge is to intelligently select the more appropriate low-level

heuristic (meme) within a set at each decision point during the search.

Adaptive operator selection: A related recent research direction, again within evo-

lutionary algorithms, has been termed adaptive operator selection. Its goal is to

design on-line strategies able to autonomously select between different variation

operators. As discussed in (Maturana et al, 2009), adaptive operator selection ap-

proaches contain two main mechanisms: credit assignment, which defines the reward

to be assigned to an operator (according to its quality) after it has been applied,

and operator selection that selects one operator to be applied according to previ-

ously computed operator qualities. Several approaches for implementing these two

mechanisms(and their combination), have been proposed (Fialho et al, 2008, 2009;

Maturana and Saubion, 2008; Maturana et al, 2009, 2010).

Adaptive operator selection is closely related to heuristic selection based on per-

turbative heuristics (section 4.2), again the difference lies in the population-based

nature of the former. The relationship with adaptive memetic algorithms is also

clear, although in adaptive operator selection the goal is to select between different

variation operators (mutation and recombination) instead of local-searchers, but

the challenges identified by the credit assignment and operator selection compo-

nents are present in both.

Reactive search: is an on-line methodology that advocates the integration of sub-

symbolic machine learning techniques into search heuristics for solving complex

optimisation problems (Battiti and Brunato, 2007; Battiti and Protasi, 1997; Bat-

titi, 1996). The machine learning component acts on top of the search heuristic,

in order to let the algorithm self-tune its operating parameters during the search

operation. The learning component is implemented as a reactive feedback scheme

that uses the past history of the search to increase its efficiency and efficacy. These

ideas have been mainly applied to the Tabu search meta-heuristic.

Variable neighborhood search: Although generally not including an adaptive mech-

anism, Variable Neighbourhood search (VNS) (Mladenovic and Hansen, 1997), is

related to heuristic selection based on perturbative heuristics in that they exploit

the search power of multiple neighbourhoods. VNS systematically switches neigh-

bourhoods in a predefined sequence so that the search can explore increasingly

distant neighbourhoods of the current solution. Therefore, we can say that VNS

is a high-level pre-designed heuristic that controls the behaviour of several neigh-

bourhood structures.

Algorithm Portfolios: First proposed in (Huberman et al, 1997), algorithm portfo-

lios represent an an alternative way of automating the design of search techniques.

They are designed following the standard practice in economics to obtain different

return-risk profiles in the stock market by combining different stocks. An algorithm

41

portfolio would run different algorithms concurrently in a time sharing manner, by

allocating a fraction of the total CPU cycles to each of them. The first algorithm

to finish reports the solution and determines the completion time of the portfo-

lio, while the other algorithms are immediately stopped. Dynamic portfolios, that

include online learning, have been considered in (Gagliolo and Schmidhuber, 2006).

7 Discussion and future work

The defining feature of hyper-heuristics is that they operate on a search space of

heuristics rather than directly on a search space of problem solutions. This feature

provides the potential for increasing the level of generality of search methodologies.

We found several independent realisations of this idea over the years (since the early

60s) within computer science, artificial intelligence and operational research. The term

hyper-heuristic, however, is relatively new; it was first used in 1997 to describe a pro-

tocol that combines several Artificial Intelligence methods in the context of automated

theorem proving. Some years later, in 2000, it was independently used to describe

‘heuristics to choose heuristics’ in the context of combinatorial optimisation. We iden-

tified two main broad classes of approaches to the challenge of automating the design of

heuristics; namely: heuristic selection and heuristic generation. In this article, we have

covered both the intellectual roots, and the state-of-the-art of these methodologies.

Heuristic generation approaches provide richer heuristic search spaces, and thus free-

dom to create new methodologies for solving the underlying combinatorial problems.

However, they are more difficult to implement, when compared to their counterpart

(heuristic selection methodologies) since they require the decomposition of existing

heuristics, and the design of an appropriate framework. We have further categorised

hyper-heuristics according to whether they are based on constructive or perturbative

low-level heuristics. These categories describe current research trends, however, the pos-

sibilities are open for the exploration of hybrid approaches. An additional orthogonal

criterion for classifying hyper-heuristics is the source of the feedback during the learn-

ing process, which can be either one instance (online approaches) or many instances of

the underlying problem (offline approaches). Both online and offline approaches are po-

tentially useful and therefore worth investigating. Although having a reusable method

will increase the speed of solving new instances of a given problem, using online (or

disposable) methods can have other advantages. In particular, searching over a space

of heuristics may be more effective than directly searching the underlying problem

space, as heuristics may provide an advantageous search space structure. Moreover, in

newly encountered problems there may not be a set of related instances on which to

train offline hyper-heuristic methods. Machine learning is a well established artificial

intelligence sub-field with a wealth of proven tools. There is still plenty of room for

exploring the suitability of these techniques for automating the design of heuristics.

Clearly, there is much ground for further research within hyper-heuristics. The chal-

lenge of fully automating the design of search methodologies is still far from fulfilled.

From our detailed study of the state of the art, we believe it is worth devoting research

efforts to the following hyper-heuristic themes:

Generalisation: By far the biggest challenge is to develop methodologies that work

well across several domains. This is the main motivation behind hyper-heuristic re-

search, and it impinges upon all the different classes of approaches: heuristic selec-

tion and generation; and online and offline techniques. Further research may be di-

42

rected to the incorporation of meta-learning ideas (i.e. clustering, data-mining, fea-

ture extraction and classification techniques) , the development of effective search

strategies on heuristic spaces, and the exploration of roboust adaptive and self-

adaptive techniques (i.e. fuzzy, reinforcement learning, and co-evolutionary ap-

proaches). Research into common representations or description formats that unify

different but related problems is also relevant for automation and generalisation of

approaches.

Foundational studies: Thus far, little progress has been made to enhance our under-

standing of hyper-heuristic approaches. Initial efforts have been devoted to under-

standing the structure of heuristic search spaces. Research in this direction would

be relevant to designing efficient hyper-heuristics. Understanding the structure of

solution spaces may also be relevant for heuristic selection and generation. Indeed,

identifying relevant features that characterise problems or classes of problems is of

outmost relevance for automated heuristic design, as it is the matching between

these characteristics and the performance of heuristic algorithms or their compo-

nents. The ‘No Free Lunch Theorem for Optimisation’ (Wolpert and Macready,

1997) states a limit on the ‘Holy Grail’ of Artificial Intelligence research to find

a single algorithm to solve all problems. However, hyper-heuristic research is con-

cerned with classes of related problems and, in this context, the literature shows

that advances can be achieved, but how far this advances can go (i.e. the determi-

nation of the boundaries of automated intelligent systems design) is very much an

open question.

Distributed, agent-based and cooperative approaches: The use of distributed

and cooperative algorithms offers several possibilities within hyper-heuristic re-

search. On one hand, the role of cooperation between low-level heuristics within

a hyper-heuristic framework is worth exploring. Since different low-level heuristics

have different strengths and weakness, cooperation can allow synergies between

them. On the other hand, we can envisage multiple, independent hyper-heuristic

solvers that act collaboratively or co-operatively in some manner to implement

group or consensus strategies in order, for example, to select the low-level heuristic

to apply next.

Multi-criteria, multi-objective and dynamic problems: So far, hyper-heuristics

have been mainly applied to single objective and static problems. Since real-world

scenarios are invariably multi-criteria and time-changing, further developments in

this theme are required. There is probably more to be gained in applying adap-

tive approaches, such as hyper-heuristics, to dynamic problems than to their static

counterparts. The multi-objective context, also offers interesting challenges for the

automation of heuristic selection and generation. Moreover, the design of adap-

tive search methodologies can in it self be considered as a multi-objective design

problem.

Generic issues: None of the themes discussed above should be taken in isolation from

each other; indeed, they cut across and feed into each other. Other essential issues

within the context of automatic heuristic design include Robustness (the capabil-

ity to change in order to cope with unforeseen events); Ease of Use (interaction

with the user represents an absolutely crucial feature in the uptake of decision

support systems); Hybridisation (drawing on the strengths and capabilities of a

range of methods, approaches and advances); and Problem Decomposition (in-

telligently breaking up larger problems into smaller, easier to handle sub-problems

and then dealing with each sub-problem in turn).

43

As it was discussed in section 6 several communities have been working in related

research themes and sharing common goals. However, there is still little interaction

between them. Much is to be gained from a greater awareness of the achievements in

various cross-disciplinary approaches; opportunities would open for extension to both

new problem domains and new methodologies through cross-fertilisation of these ideas.

Hyper-heuristic research has the potential of bringing together promising ideas in the

fields of modern search heuristics and machine learning, with knowledge (in the form

of problem-specific heuristics) accumulated over the years in the field of operational

research. The overall aim is to solve complex real-life combinatorial optimisation prob-

lems in a more general fashion. The ultimate goal is is to produce reusable technologies

to underpin future research into automated heuristic design to facilitate systems which

can work with users to home in on high quality solutions to problems.

References

Adenso-Diaz B, Laguna M (2006) Fine-tuning of algorithms using fractional experi-

mental design and local search. Operations Research 54(1):99–114

Ahmadi S, Barrone P, Cheng P, Burke EK, Cowling P, McCollum B (2003) Per-

turbation based variable neighbourhood search in heuristic space for examination

timetabling problem. In: Proceedings of Multidisciplinary International Scheduling:

Theory and Applications (MISTA 2003), Nottingham, August 13-16, pp 155–171

Aickelin U, Burke EK, Li J (2009) An evolutionary squeaky wheel optimization ap-

proach to personnel scheduling. IEEE Transactions on Evolutionary Computation

13(2):433–443

Allen S, Burke EK, Hyde MR, Kendall G (2009) Evolving reusable 3d packing heuristics

with genetic programming. In: Proceedings of the ACM Genetic and Evolutionary

Computation Conference (GECCO ’09), Montreal, Canada, pp 931–938

Antunes CH, Lima P, Oliveira E, Costa DF (2009) A multi-objective simulated an-

nealing approach to reactive power compensation. In: Proceedings of the 8th Meta-

heuristic International Conference

Asmuni H, Burke EK, Garibaldi JM (2005) Fuzzy multiple heuristic ordering for course

timetabling. In: Proceedings of the 5th United Kingdom Workshop on Computational

Intelligence (UKCI 2005), London, UK, pp 302–309

Ayob M, Kendall G (2003) A monte carlo hyper-heuristic to optimise component place-

ment sequencing for multi head placement machine. In: Proceedings of the Interna-

tional Conference on Intelligent Technologies (InTech’03), Chiang Mai, Thailand, pp

132–141

Bader-El-Din MB, Poli R (2007) Generating sat local-search heuristics using a gp

hyper-heuristic framework. In: LNCS 4926. Proceedings of the 8th International

Conference on Artifcial Evolution, pp 37–49

Bader-El-Din MB, Poli R (2008) An incremental approach for improving local search

heuristics. In: LNCS 4972. Proceedings of the 8th European Conference on Evolu-

tionary Computation in Combinatorial Optimization (EvoCOP’08), Naples, Italy,

pp 194–205

Bader-El-Din MB, Poli R (2009) Grammar-based genetic programming for timetabling.

In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC’09),

Trondheim, Norway, pp 2532–2539

44

Bai R, Kendall G (2005) An investigation of automated planograms using a simu-

lated annealing based hyper-heuristics. In: Ibaraki T, Nonobe K, Yagiura M (eds)

Metaheuristics: Progress as Real Problem Solver - (Operations Research/Computer

Science Interface Serices, Vol.32), Springer, pp 87–108

Bai R, Kendall G (2008) A model for fresh produce shelf-space allocation and inven-

tory management with freshness-condition-dependent demand. INFORMS Journal

on Computing 20(1):78–85

Bai R, Blazewicz J, Burke EK, Kendall G, McCollum B (2007a) A simulated annealing

hyper-heuristic methodology for flexible decision support. Tech. Rep. NOTTCS-TR-

2007-8, School of CSiT, University of Nottingham

Bai R, Burke EK, Kendall G, McCollum B (2007b) Memory length in hyper-heuristics:

An empirical study. In: Proceedings of the 2007 IEEE Symposium on Computational

Intelligence in Scheduling (CISched2007), Hilton Hawaiian Village, Honolulu, Hawaii,

USA

Bai R, Burke EK, Kendall G (2008) Heuristic,meta-heuristic and hyper-heuristic ap-

proaches for fresh produce inventory control and shelf space allocation. Journal of

the Operational Research Society 59:1387 – 1397

Battiti R (1996) Reactive search: Toward self–tuning heuristics. In: Rayward-Smith

VJ, Osman IH, Reeves CR, Smith GD (eds) Modern Heuristic Search Methods,

John Wiley & Sons Ltd., Chichester, pp 61–83

Battiti R, Brunato M (2007) Reactive search: Machine learning for memory-based

heuristics. In: Gonzalez TF (ed) Approximation Algorithms and Metaheuristics, Tay-

lor and Francis Books (CRC Press), Washington, DC, chap 21, pp 1–17

Battiti R, Protasi M (1997) Reactive search, a history-sensitive heuristic for MAX-SAT.

ACM Journal of Experimental Algorithms 2:2

Bhanu SMS, Gopalan N (2008) A hyper-heuristic approach for efficient resource

scheduling in grid. International Journal of Computers, Communications & Control

III(3):249–258

Biazzini M, Banhelyi B, Montresor A, Jelasity M (2009) Distributed hyper-heuristics

for real parameter optimization. In: GECCO ’09: Proceedings of the 11th Annual

conference on Genetic and evolutionary computation, ACM, New York, NY, USA,

pp 1339–1346, DOI http://doi.acm.org/10.1145/1569901.1570081

Bilgin B, Ozcan E, Korkmaz EE (2006) An experimental study on hyper-heuristics

and final exam scheduling. In: Proc. of the International Conference on the Practice

and Theory of Automated Timetabling (PATAT’06), pp 123–140

Birattari M (2004) The problem of tuning metaheuristics as seen from a machine

learning perspective. PhD thesis, Universite Libre de Bruxelles

Birattari M, Stutzle T, Paquete L, Varrentrapp K (2002) A racing algorithm for config-

uring metaheuristics. In: GECCO 2002: Proceedings of the Genetic and Evolutionary

Computation Conference, New York, USA, 9-13 July 2002, Morgan Kaufmann, pp

11–18

Burke EK, Bykov Y (2008) A late acceptance strategy in hill-climbing for exam

timetabling problems. In: Conference on the Practice and Theory of Automated

Timetabling (PATAT 2008)

Burke EK, Hart E, Kendall G, Newall J, Ross P, Schulenburg S (2003a) Hyper-

heuristics: An emerging direction in modern search technology. In: Glover F, Kochen-

berger G (eds) Handbook of Metaheuristics, Kluwer, pp 457–474

Burke EK, Kendall G, Soubeiga E (2003b) A tabu-search hyperheuristic for timetabling

and rostering. Journal of Heuristics 9(6):451–470

45

Burke EK, Dror M, Petrovic S, Qu R (2005a) The Next Wave in Computing, Opti-

mization, and Decision Technologies, Springer, chap Hybrid Graph Heuristics within

a Hyper-heuristic Approach to Exam Timetabling Problems, pp 79–91

Burke EK, Kendall G, Landa-Silva JD, O’Brien R, Soubeiga E (2005b) An ant algo-

rithm hyperheuristic for the project presentation scheduling problem. In: Proceed-

ings of the 2005 IEEE Congress on Evolutionary Computation, Edinburgh, Scotland,

vol 3, pp 2263–2270

Burke EK, Landa-Silva JD, Soubeiga E (2005c) Meta-heuristics: Progress as Real Prob-

lem Solvers, Springer, chap Multi-objective Hyper-heuristic Approaches for Space

Allocation and Timetabling, pp 129–158,

Burke EK, Hyde MR, Kendall G (2006a) Evolving bin packing heuristics with genetic

programming. In: Runarsson T, Beyer HG, Burke E, JMerelo-Guervos J, Whitley D,

Yao X (eds) LNCS 4193, Proceedings of the 9th International Conference on Parallel

Problem Solving from Nature (PPSN’06), Reykjavik, Iceland, pp 860–869

Burke EK, Petrovic S, Qu R (2006b) Case based heuristic selection for timetabling

problems. Journal of Scheduling 9(2):115–132

Burke EK, Hyde MR, Kendall G, Woodward J (2007a) Automatic heuristic genera-

tion with genetic programming: Evolving a jack-of-all-trades or a master of one. In:

Proceedings of the 9th ACM Genetic and Evolutionary Computation Conference

(GECCO’07), London, UK., pp 1559–1565

Burke EK, Hyde MR, Kendall G, Woodward J (2007b) The scalability of evolved on

line bin packing heuristics. In: Proceedings of the IEEE Congress on Evolutionary

Computation (CEC’07), Singapore, pp 2530–2537

Burke EK, McCollum B, Meisels A, Petrovic S, Qu R (2007c) A graph-based hyper-

heuristic for educational timetabling problems. European Journal of Operational

Research 176:177–192

Burke EK, Kendall G, sır MM, Ozcan E (2008) Monte carlo hyper-heuristics for ex-

amination timetabling. In: Conference on the Practice and Theory of Automated

Timetabling (PATAT 2008)

Burke EK, Hyde M, Kendall G, Ochoa G, Ozcan E, Woodward J (2009a) Exploring

hyper-heuristic methodologies with genetic programming. In: Mumford C, Jain L

(eds) Computational Intelligence: Collaboration, Fusion and Emergence, Intelligent

Systems Reference Library, Springer, pp 177–201

Burke EK, Hyde M, Kendall G, Ochoa G, Ozcan E, Woodward J (2009b) Handbook

of Metaheuristics, Springer, chap A Classification of Hyper-heuristic Approaches.

International Series in Operations Research & Management Science

Burke EK, Hyde M, Kendall G, Woodward J (2010) A genetic programming hyper-

heuristic approach for evolving two dimensional strip packing heuristics. IEEE Trans-

actions on Evolutionary Computation (to appear)

Cano-Belman J, and J Bautista RZRM (2010) A scatter search based hyper-heuristic

for sequencing a mixed-model assembly line. Journal of Heuristics (to appear)

Cerny V (1985) Thermodynamical approach to the traveling salesman problem: An

efficient simulation algorithm. Journal of Optimization Theory and Applications

45(1):41–51

Chakhlevitch K, Cowling P (2005) Choosing the fittest subset of low level heuristics in a

hyperheuristic framework. In: Proceedings of 5th European Conference on Evolution-

ary Computation in Combinatorial Optimization (EvoCOP2005), Springer, Lecture

Notes in Computer Science, vol 3448, pp 25–33

46

Chakhlevitch K, Cowling PI (2008) Hyperheuristics: Recent developments. In: Cotta

C, Sevaux M, Sorensen K (eds) Adaptive and Multilevel Metaheuristics, Studies in

Computational Intelligence, vol 136, Springer, pp 3–29

Chen PC, Kendall G, Vanden-Berghe G (2007) An ant based hyper-heuristic for the

travelling tournament problem. In: Proceedings of IEEE Symposium of Computa-

tional Intelligence in Scheduling (CISched 2007), Hawaii, pp 19–26

Cowling P, Chakhlevitch K (2003) Hyperheuristics for managing a large collection of

low level heuristics to schedule personnel. In: Proceedings of the IEEE Congress on

Evolutionary Computation (CEC2003), IEEE Computer Society Press, Canberra,

Australia, pp 1214–1221

Cowling P, Kendall G, Soubeiga E (2000) A hyperheuristic approach for scheduling

a sales summit. In: Selected Papers of the Third International Conference on the

Practice And Theory of Automated Timetabling, PATAT 2000, Springer, Konstanz,

Germany, Lecture Notes in Computer Science, pp 176–190

Cowling P, Kendall G, Soubeiga E (2001) A parameter-free hyperheuristic for schedul-

ing a sales summit. In: Proceedings of the 4th Metaheuristic International Confer-

ence, pp 127–131

Cowling P, Kendall G, Han L (2002a) An investigation of a hyperheuristic genetic

algorithm applied to a trainer scheduling problem. In: Proceedings of the Congress

on Evolutionary Computation (CEC2002), Hilton Hawaiian Village Hotel, Honolulu,

Hawaii, USA, pp 1185–1190

Cowling P, Kendall G, Soubeiga E (2002b) Hyperheuristics: A tool for rapid prototyp-

ing in scheduling and optimisation. In: Cagoni S, Gottlieb J, Hart E, Middendorf M,

Goenther R (eds) Applications of Evolutionary Computing: Proceeding of Evo Work-

shops 2002, Springer-Verlag, Kinsale, Ireland, Lecture Notes in Computer Science,

vol 2279, pp 1–10

Coy S, Golden BL, Runger GC, Wasil EA (2001) Using experimental design to find

effective parameter settings for heuristics. J Heuristics 7(1):77–97

Crainic T, Toulouse M (2003) Parallel strategies for meta-heuristics. In: Glover F,

Kochenberger G (eds) Handbook in Meta-heuristics, Kluwer Academic Publishers,

pp 475–513

Crowston WB, Glover F, Thompson GL, Trawick JD (1963) Probabilistic and para-

metric learning combinations of local job shop scheduling rules. ONR Research mem-

orandum, GSIA, Carnegie Mellon University, Pittsburgh -(117)

Davis L (1989) Adapting operator probabilities in genetic algorithms. In: Schaffer

JD (ed) Proceedings of the 3rd International Conference on Genetic Algorithms

(ICGA89), San Francisco, pp 61–69

Deb K, Pratap A, Agarwal S, Meyarivan T (2000) A fast elitist multi-objective genetic

algorithm: Nsga-ii. IEEE Transactions on Evolutionary Computation 6:182–197

Denzinger J, Fuchs M, Fuchs M (1997) High performance ATP systems by combining

several ai methods. In: Proceedings of the Fifteenth International Joint Conference

on Artificial Intelligence (IJCAI 97), pp 102–107

DiGaspero L, Schaerf A (2007) Easysyn++: A tool for automatic synthesis of stochas-

tic local search algorithms. In: Stutzle T, Birattari M, Hoos H (eds) LNCS 4638.

Proceedings of the International Workshop on Engineering Stochastic Local Search

Algorithms (SLS 2007), pp 177–181

Dimopoulos C, Zalzala AMS (2001) Investigating the use of genetic programming

for a classic one-machine scheduling problem. Advances in Engineering Software

32(6):489–498

47

Dorndorf U, Pesch E (1995) Evolution based learning in a job shop scheduling envi-

ronment. Computers and Operations Research 22(1):25–40

Dowsland KA, Soubeiga E, Burke EK (2007) A simulated annealing hyper-heuristic for

determining shipper sizes. European Journal of Operational Research 179(3):759–774

Drechsler R, Becker B (1995) Learning heuristics by genetic algorithms. In: Proceedings

of the ASP Design Automation Conference, Mukuhari, Japan, pp 349–352

Drechsler R, Gockel N, Becker B (1996) Learning heuristics for obdd minimization

by evolutionary algorithms. In: LNCS 1141. Proceedings of the 4th International

Conference on Parallel Problem Solving from Nature (PPSN’96), Berlin, Germany,

pp 730–739

Dueck G (1993) New optimization hueristics: The great deluge algorithm and the

record-to record travel. Journal of Computational Physics 104:86–92

Eiben AE, Hinterding R, Michalewicz Z (1999) Parameter control in Evolutionary

Algorithms. IEEE Transactions on Evolutionry Computation 3(2):124

Eiben AE, Michalewicz Z, Schoenauer M, Smith JE (2007) Parameter Setting in Evolu-

tionary Algorithms, Springer, chap Parameter Control in Evolutionary Algorithms,

pp 19–46

Ersoy E, Ozcan E, Etaner-Uyar AS (2007) Memetic algorithms and hyperhill-climbers.

In: Baptiste P, Kendall G, Kordon AM, Sourd F (eds) MISTA 2007: 3rd Multidisci-

plinary International Scheduling Conference: Theory and Applications, pp 159–166

Fang H, Ross P, Corne D (1993) A promising genetic algorithm approach to job shop

scheduling, rescheduling, and open-shop scheduling problems. In: Forrest S (ed) Fifth

International Conference on Genetic Algorithms, Morgan Kaufmann, San Mateo, pp

375–382

Fang H, Ross P, Corne D (1994) A promising hybrid ga/ heuristic approach for open-

shop scheduling problems. In: Cohn A (ed) Eleventh European Conference on Arti-

ficial Intelligence, John Wiley & Sons

Feo TA, Resende MGC (1995) Greedy randomized adaptive search procedures. Journal

of Global Optimization 6:109–133

Fialho A, Costa L, Schoenauer M, Sebag M (2008) Extreme value based adaptive

operator selection. In: Proceedings of the 10th international conference on Parallel

Problem Solving from Nature, Springer-Verlag, Berlin, Heidelberg, pp 175–184

Fialho A, Da Costa L, Schoenauer M, Sebag M (2009) Extreme: dynamic multi-armed

bandits for adaptive operator selection. In: GECCO ’09: Proceedings of the 11th An-

nual Conference Companion on Genetic and Evolutionary Computation Conference,

ACM, New York, NY, USA, pp 2213–2216

Fisher H, Thompson GL (1961) Probabilistic learning combinations of local job-shop

scheduling rules. In: Factory Scheduling Conference, Carnegie Institue of Technology

Fisher H, Thompson GL (1963) Probabilistic learning combinations of local job-shop

scheduling rules. In: Muth JF, Thompson GL (eds) Industrial Scheduling, Prentice-

Hall, Inc, New Jersey, pp 225–251

Freisleben B, Hartfelder M (1993) Optimization of genetic algorithms by genetic al-

gorithms. In: Albrecht RF, Steele NC, Reeves CR (eds) Artificial Neural Nets and

Genetic Algorithms, Springer Verlag, Wien, pp 392–399

Fukunaga AS (2002) Automated discovery of composite sat variable-selection heuris-

tics. In: Eighteenth national conference on Artificial intelligence, Edmonton, Alberta,

Canada, pp 641–648

Fukunaga AS (2004) Evolving local search heuristics for SAT using genetic program-

ming. In: et al KD (ed) LNCS 3103. Proceedings of the ACM Genetic and Evolu-

48

tionary Computation Conference (GECCO ’04), Springer-Verlag, Seattle, WA, USA,

pp 483–494

Fukunaga AS (2008) Automated discovery of local search heuristics for satisfiability

testing. Evolutionary Computation (MIT Press) 16(1):31–1

Gagliolo M, Schmidhuber J (2006) Dynamic algorithm portfolios. In: Proceedings AI

and MATH 06, Ninth International Symposium on Artificial Intelligence and Math-

ematics, Fort Lauderdale, Florida

Garrido P, Castro C (2009) Stable solving of cvrps using hyperheuristics. In: Genetic

and Evolutionary Computation Conference (GECCO’09), ACM, Montreal, Canada,

pp 255–262

Garrido P, Riff MC (2007a) Collaboration between hyperheuristics to solve strip-

packing problems. In: 12th International Fuzzy Systems Association World Congress,

Proceedings, Springer, Lecture Notes in Computer Science, vol 4529, pp 698–707

Garrido P, Riff MC (2007b) An evolutionary hyperheuristic to solve strip-packing prob-

lems. In: Intelligent Data Engineering and Automated Learning - IDEAL 2007 Pro-

ceedings, Springer, Lecture Notes in Computer Science, vol 4881, pp 406–415

Garrido P, Riff MC (2010) Dvrp: A hard dynamic combinatorial optimisation problem

tackled by an evolutionary hyper-heuristic. Journal of Heuristics (to appear)

Geiger CD, Uzsoy R, Aytug H (2006) Rapid modeling and discovery of priority dis-

patching rules: An autonomous learning approach. Journal of Scheduling 9(1):7–34

Glover F (1986) Future paths for integer programming and links to artificial intelli-

gence. Comput Oper Res 13(5):533–549

Goldberg DE, Korb B, Deb K (1990) Messy genetic algorithms: Motivation, analysis,

and first results. Complex Systems 3(5):493–530

Gratch J, Chien S (1996) Adaptive problem-solving for large-scale scheduling problems:

a case study. Journal of Artificial Intelligence Research 4:365–396

Gratch J, Chien S, DeJong G (1993) Learning search control knowledge for deep space

network scheduling. In: Proceedings of the Tenth International Conference on Ma-

chine Learning, Amherst, MA, pp 135–142

Grefenstette JJ (1986) Optimization of control parameters for genetic algorithms. IEEE

Transactions on Systems, Man, and Cybernetics SMC-16(1):122–128

Han L, Kendall G (2003) Guided operators for a hyper-heuristic genetic algorithm.

In: Proceedings of AI-2003: Advances in Artificial Intelligence. The 16th Australian

Conference on Artificial Intelligence (AI 03), Perth, Australia, pp 807–820

Hart E, Ross P (1998) A heuristic combination method for solving job-shop scheduling

problems. In: Eiben AE, Back T, Schoenauer M, Schwefel HP (eds) Parallel Problem

Solving from Nature V, Springer-Verlag, Lecture Notes in Computer Science, vol

1498, pp 845–854

Hart E, Ross P, Nelson JAD (1998) Solving a real-world problem using an evolving

heuristically driven schedule builder. Evolutionary Computing 6(1):61–80

Hart W, Krasnogor N, Smith J (eds) (2004) Recent Advances in Memetic Algorithms.

Springer, Berlin, Heidelberg, New York

Ho NB, Tay JC (2005) Evolving dispatching rules for solving the flexible job-shop prob-

lem. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC’05),

Edinburgh, UK, pp 2848–2855

Horvitz E, Ruan Y, Gomes CP, Kautz HA, Selman B, Chickering DM (2001) A bayesian

approach to tackling hard computational problems. In: UAI ’01: Proceedings of the

17th Conference in Uncertainty in Artificial Intelligence, Morgan Kaufmann Pub-

lishers Inc., San Francisco, CA, USA, pp 235–244

49

Huberman BA, Lukose RM, , Hogg T (1997) An economics approach to hard compu-

tational problems. Science 275:51–54

Hutter F, Hoos HH, Stutzle T (2007) Automatic algorithm configuration based on local

search. In: AAAI, AAAI Press, pp 1152–1157

Hutter F, Hoos H, Leyton-Brown K, Stutzle T (2009) Paramils: An automatic algo-

rithm configuration framework. Journal of Artificial Intelligence Research (JAIR)

36:267306

Jakob W (2002) HyGLEAM – an approach to generally applicable hybridization of

evolutionary algorithms. In: Parallel Problem Solving from Nature - PPSN VII,

Springer, Berlin, pp 527–536

Jakob W (2006) Towards an adaptive multimeme algorithm for parameter optimisation

suiting the engineers’ needs. In: Parallel Problem Solving from Nature - PPSN IX,

9th International Conference, Reykjavik, Iceland, September 9-13, 2006, Procedings,

Springer, Lecture Notes in Computer Science, vol 4193, pp 132–141

Jakobovic D, Jelenkovic L, Budin L (2007) Genetic programming heuristics for multiple

machine scheduling. In: LNCS 4445. Proceedings of the European Conference on

Genetic Programming (EUROGP’07, Valencia, Spain, pp 321–330

Joslin D, Clements DP (1999) ”squeaky wheel” optimization. Journal of Artificial In-

telligence Research 10:353–373

Julstrom BA (1995) What have you done for me lately? Adapting operator probabilities

in a steady-state genetic algorithm. In: Eshelman L (ed) Proceedings of the Sixth

International Conference on Genetic Algorithms, Morgan Kaufmann, San Francisco,

CA, pp 81–87

Kaelbling LP, Littman ML, Moore AW (1996) Reinforcement learning: a survey. Jour-

nal of Artificial Intelligence Research 4:237–285

Keller RE, Poli R (2007a) Cost-benefit investigation of a genetic-programming hyper-

heuristic. In: Proceedings of the 8th International Conference on Artifcial Evolution,

Tours, France, pp 13–24

Keller RE, Poli R (2007b) Linear genetic programming of parsimonious metaheuris-

tics. In: Proceedings of the IEEE Congress on Evolutionary Computation (CEC’07),

Singapore, pp 4508–4515

Keller RE, Poli R (2008a) Self-adaptive hyperheuristic and greedy search. In: Wang

J (ed) 2008 IEEE World Congress on Computational Intelligence (WCCI’08), Hong

Kong

Keller RE, Poli R (2008b) Subheuristic search and scalability in a hyperheuristic. In:

Proceedings of the 10th annual conference on Genetic and evolutionary computation

(GECCO’08), pp 609–610

Keller RE, Poli R (2008c) Toward subheuristic search. In: Wang J (ed) 2008 IEEE

World Congress on Computational Intelligence (WCCI’08), Hong Kong

Kendall G, Mohamad M (2004a) Channel assignment in cellular communication us-

ing a great deluge hyper-heuristic. In: Proceedings of the 2004 IEEE International

Conference on Network (ICON2004), Singapore, pp 769–773

Kendall G, Mohamad M (2004b) Channel assignment optimisation using a hyper-

heuristic. In: Proceedings of the 2004 IEEE Conference on Cybernetic and Intelligent

Systems (CIS2004), Singapore, pp 790–795

Kenyon C (1996) Best-fit bin-packing with random order. In: Proceedings of the Sev-

enth Annual ACM-SIAM Symposium on Discrete Algorithms, pp 359–364

Kirkpatrick S, Gelatt CD, Vecchi MP (1983) Optimization by simulated annealing.

Science 220:671–680

50

Koza JR (1992) Genetic programming: on the programming of computers by means of

natural selection. The MIT Press, Boston, Massachusetts

Koza JR (1994) Genetic programming II: automatic discovery of reusable programs.

The MIT Press, Cambridge, Massachusetts

Koza JR, Poli R (2005) Genetic programming. In: Burke EK, Kendall G (eds) Search

Methodologies: Introductory Tutorials in Optimization and Decision Support Tech-

niques, Kluwer, Boston, pp 127–164

Krasnogor N (2004) Self generating metaheuristics in bioinformatics: The proteins

structure comparison case. Genetic Programming and Evolvable Machines 5(2):181–

201

Krasnogor N, Gustafson S (2002) Toward truly “memetic” memetic algorithms: dis-

cussion and proofs of concept. In: DCorne, Fogel G, Hart W, Knowles J, Krasno-

gor N, Roy R, Smith J, Tiwari A (eds) Advances in Nature-Inspired Computation:

The PPSN VII Workshops, PEDAL (Parallel, Emergent & Distributed Architectures

Lab), University of Reading, Reading, UK, pp 9–10

Krasnogor N, Gustafson S (2004) A study on the use of “self-generation” in memetic

algorithms. Natural Computing 3(1):53–76

Krasnogor N, Smith JE (2000) A memetic algorithm with self-adaptive local search:

TSP as a case study. In: Proceedings of the 2000 Genetic and Evolutionary Compu-

tation Conference, Morgan Kaufmann

Krasnogor N, Smith JE (2001) Emergence of profitable search strategies based on a

simple inheritance mechanism. In: Proceedings of the 2001 Genetic and Evolutionary

Computation Conference, Morgan Kaufmann

Krasnogor N, Blackburne BP, Burke EK, Hirst J (2002) Multimeme algorithms for

protein structure. In: In: Proceedings of the Parallel Problem Solving from Nature

VII. Lecture Notes in Computer Science, Springer-Verlag, pp 769–778

Kumar R, Joshi AH, Banka KK, Rockett PI (2008) Evolution of hyperheuristics for

the biobjective 0/1 knapsack problem by multiobjective genetic programming. In:

Proceedings of the 10th ACM conference on Genetic and evolutionary computation

(GECCO’08), Atlanta, GA, USA, pp 1227–1234

Kumar R, Kumar Bal B, Rockett PI (2009) Multiobjective genetic programming ap-

proach to evolving heuristics for the bounded diameter minimum spanning tree prob-

lem. In: Proceedings of the ACM Genetic and Evolutionary Computation Conference

(GECCO ’09), Montreal, Canada, pp 309–316

Laguna M, Martı R (2003) Scatter Search. Kluwer, Boston

Leake DB (1996) Case Based Reasoning: Experiences, Lessons, and Future Directions.

AAI Press/MIT Press

Leyton-Brown K, Nudelman E, Shoham Y (2002) Learning the empirical hardness

of optimization problems: The case of combinatorial auctions. In: Lecture Notes in

Computer Science, Springer, vol 2470, pp 556–572

Lobo F, Goldberg D (1997) Decision making in a hybrid genetic algorithm. In: Proc.

IEEE International Conference on Evolutionary Computation, pp 121–125

Lobo FG, Lima CF, Michalewicz Z (eds) (2007) Parameter Setting in Evolutionary

Algorithms, Studies in Computational Intelligence, vol 54. Springer

Lokketangen A, Olsson R (2010) Generating meta-heuristic optimization code using

adate. Journal of Heuristics (to appear)

Lundy M, Mees A (1986) Convergence of an annealing algorithm. Mathematical Pro-

gramming 34:111–124

51

Marın-Blazquez JG, Schulenburg S (2007) A hyper-heuristic framework with XCS:

Learning to create novel problem-solving algorithms constructed from simpler algo-

rithmic ingredients. In: IWLCS, Springer, Lecture Notes in Computer Science, vol

4399, pp 193–218

Maturana J, Saubion F (2008) A compass to guide genetic algorithms. In: Proceed-

ings of the 10th international conference on Parallel Problem Solving from Nature,

Springer-Verlag, Berlin, Heidelberg, pp 256–265

Maturana J, Fialho A, Saubion F, Schoenauer M, Sebag M (2009) Extreme compass

and dynamic multi-armed bandits for adaptive operator selection. In: Proc. IEEE

Congress on Evolutionary Computation CEC ’09, pp 365–372

Maturana J, Lardeux F, Saubion F (2010) Autonomous operator management for

evolutionary algorithms. Journal of Heuristics (to appear)

Meignan D, Koukam A, Creput JC (2010) Coalition-based metaheuristic: A self-

adaptive metaheuristic using reinforcement learning and mimetism. Journal of

Heuristics (to appear)

Minton S (1996) Automatically configuring constraint satisfaction problems: a case

study. Constraints 1(1):7–43

Mladenovic N, Hansen P (1997) Variable neighborhood search. Computers and Oper-

ations Research 24(11):1097–1100

Nareyek A (2001) An empirical analysis of weight-adaptation strategies for neighbor-

hoods of heuristics. In: Metaheuristic International Conference MIC’2001, Porto,

Portugal, pp 211–215

Nareyek A (2003) Choosing search heuristics by non-stationary reinforcement learning.

In: Resende MGC, de Sousa JP (eds) Metaheuristics: Computer Decision-Making,

Kluwer, chap 9, pp 523–544

Neri F, Tirronen V, Karkkainen T, Rossi T (2007a) Fitness diversity based adaptation

in multimeme algorithms: A comparative study. In: IEEE Congress on Evolutionary

Computation, IEEE, pp 2374–2381

Neri F, Toivanen J, Cascella GL, Ong YS (2007b) An adaptive multimeme algorithm for

designing HIV multidrug therapies. IEEE/ACM Trans Comput Biology Bioinform

4(2):264–278

Norenkov I, Goodman E (1997) Solving scheduling problems via evolutionary methods

for rule sequence optimization. 2nd World Conference on soft Computing, WSC2

Ochoa G, Qu R, Burke EK (2009a) Analyzing the landscape of a graph based hyper-

heuristic for timetabling problems. In: Proceedings of Genetic and Evolutionary

Computation Conference (GECCO 2009), Montreal, Canada, submitted

Ochoa G, Vaquez-Rodrıguez JA, Petrovic S, Burke EK (2009b) Dispatching rules for

production scheduling: a hyper-heuristic landscape analysis. In: Proceedings of the

IEEE Congress on Evolutionary Computation (CEC 2009), Montreal, Norway, ac-

cepted

Oltean M (2005) Evolving evolutionary algorithms using linear genetic programming.

Evolutionary Computation 13(3):387–410

Oltean M, Dumitrescu D (2004) Evolving TSP heuristics using multi expression pro-

gramming. In: Bubak M, van Albada GD, Sloot PMA, Dongarra J (eds) LNCS

3037. Proceedings of the 4th International Conference on Computational Science

(ICCS’04), Krakow, Poland, pp 670–673

Oltean M, Grosan C (2003) Evolving evolutionary algorithms using multi expression

programming. In: Banzhaf W, Christaller T, Dittrich P, Kim JT, Ziegler J (eds)

LNAI 2801. Proceedings of the 7th European Conference on Artificial Life, Dort-

52

mund, Germany, pp 651–658

Ong YS, Keane AJ (2004) Meta-lamarckian learning in memetic algorithms. IEEE

Transactions on Evolutionry Computation 8:99–110

Ong YS, Lim MH, Zhu N, Wong KW (2006a) Classification of adaptive memetic algo-

rithms: a comparative study. IEEE Transactions on Systems, Man, and Cybernetics,

Part B 36(1):141–152

Ong YS, Lim MH, Zhu N, Wong KW (2006b) Classification of adaptive memetic algo-

rithms: a comparative study. IEEE Transactions on Systems, Man, and Cybernetics,

Part B 36(1):141–152

Ouelhadj D, Petrovic S (2008) A cooperative distributed hyper-heuristic framework for

scheduling. In: Proceedings of the IEEE International Conference on Man, Cyber-

netics, and Systems, pp 1232–1238

Ouelhadj D, Petrovic S (2009) Asynchronous cooperative hyper-heuristic search. In:

Proceedings of the International Conference on Artificial Intelligence, pp 78–84

Ouelhadj D, Petrovic S (2010) A cooperative hyper-heuristic search framework. Journal

of Heuristics (to appear)

Ozcan E, Bilgin B, Korkmaz EE (2006) Hill climbers and mutational heuristics in

hyperheuristics. In: Proceedings of the 9th International Conference on Parallel

Problem Solving from Nature (PPSN 2006), Reykjavik, Iceland, Lecture Notes in

Computer Science, vol 4193, pp 202–211

Ozcan E, Bilgin B, Korkmaz EE (2008) A comprehensive survey of hyperheuristics.

Intelligent Data Analysis 12(1):1–21

Ozcan E, Bykov Y, Birben M, Burke EK (2009) Examination timetabling using late ac-

ceptance hyper-heuristics. In: Proceedings of IEEE Congress on Evolutionary Com-

putation (CEC 2009), pp 997–1004

Pillay N (2008) An analysis of representations for hyper-heuristics for the uncapacitated

examination timetabling problem in a genetic programming system. In: Proceedings

of the 2008 Annual Conference of the South African Institute of Computer Scientists

and Information Technologists on IT Research in Developing Countries, SAICSIT

Conf. 2008, Wilderness, South Africa, October 6-8, 2008, ACM, ACM International

Conference Proceeding Series, vol 338, pp 188–192

Pillay N (2009) Evolving hyper-heuristics for the uncapacitated examination

timetabling problem. In: Proceedings of the 4th Multidisciplinary International Con-

ference on Scheduling: Theory and Applications (MISTA’09), Dublin, Ireland, pp

447–457

Pillay N, Banzhaf W (2007) A genetic programming approach to the generation of

hyper-heuristics for the uncapacitated examination timetabling problem. In: Neves J,

Santos MF, Machado J (eds) Progress in Artificial Intelligence, 13th Portuguese Con-

ference on Aritficial Intelligence, EPIA 2007 Proceedings, Springer, Lecture Notes

in Computer Science, vol 4874, pp 223–234

Pisinger D, Ropke S (2007) A general heuristic for vehicle routing problems. Computers

and Operations Research 34:2403– 2435

Poli R, Woodward JR, Burke EK (2007) A histogram-matching approach to the evo-

lution of bin-packing strategies. In: Proceedings of the Congress on Evolutionary

Computation (CEC 2007), Singapore, pp 3500–3507

Qu R, Burke EK (2009) Hybridisations within a graph based hyper-heuristic framework

for university timetabling problems. Journal of the Operational Research Society

60:1273–1285

53

Rattadilok P, Gaw A, Kwan RSK (2005) Distributed choice function hyper-heuristics

for timetabling and scheduling. In: Burke EK, Trick M (eds) The Practice and Theory

of Automated Timetabling V: Selected Papers from the 5th International Conference

on the Practice and Theory of Automated Timetabling, Lecture Notes in Computer

Science Series, vol 3616, Springer, pp 51–70

Rechenberg I (1973a) Evolution strategy: Optimization of technical systems by means

of biological evolution. Fromman-Holzboog, Stuttgart

Rechenberg I (1973b) Evolutionsstrategie: Optimierung technischer Systeme nach

Prinzipien der biologischen Evolution. Frommann-Holzboog, Stuttgart

Rice JR (1976) The algorithm selection problem. Advances in Comput 15:65118

Ridge E, Kudenko D (2007) Analyzing heuristic performance with response surface

models: prediction, optimization and robustness. In: GECCO ’07: Proceedings of

the 9th annual conference on Genetic and evolutionary computation, ACM Press,

vol 1, pp 150–157

Ross P (2005) Hyper-heuristics. In: Burke EK, Kendall G (eds) Search Methodologies:

Introductory Tutorials in Optimization and Decision Support Techniques, Springer,

chap 17, pp 529–556

Ross P, Marın-Blazquez JG (2005) Constructive hyper-heuristics in class timetabling.

In: IEEE Congress on Evolutionary Computation, IEEE, pp 1493–1500

Ross P, Hart E, Corne D (1997) Some observations about GA-based exam timetabling.

In: Burke EK, Carter MW (eds) PATAT, Springer, Lecture Notes in Computer Sci-

ence, vol 1408, pp 115–129

Ross P, Schulenburg S, Marın-Blazquez JG, Hart E (2002) Hyper-heuristics: learning

to combine simple heuristics in bin-packing problem. In: Proceedings of the Genetic

and Evolutionary Computation COnference, GECCO’02, Morgan-Kauffman

Ross P, Marın-Blazquez JG, Schulenburg S, Hart E (2003) Learning a procedure that

can solve hard bin-packing problems: A new ga-based approach to hyper-heuristics.

In: Proceedings of the Genetic and Evolutionary Computation Conference, GECCO

2003, Springer, pp 1295–1306

Ross P, Marın-Blazquez JG, Hart E (2004) Hyper-heuristics applied to class and exam

timetabling problems. In: Proceedings of the 2004 IEEE Congress on Evolutionary

Computation, IEEE Press, Portland, Oregon, pp 1691–1698

Runka A (2009) Evolving an edge selection formula for ant colony optimization. In: Pro-

ceedings of the ACM Genetic and Evolutionary Computation Conference (GECCO

’09), Montreal, Canada, pp 1075–1081

Schmiedle F, Drechsler N, Große D, Drechsler R (2002) Heuristic learning based on

genetic programming. Genetic Programming and Evolvable Machines 4:363–388

Schwefel HP (1977) Numerische Optimierung von Computer-Modellen mittels der Evo-

lutionstrategie, ISR, vol 26. Birkhaeuser, Basel/Stuttgart

Shaw P (1998) Using constraint programming and local search methods to solve vehicle

routing problems. In: Proc. of International Conference on Principles and Practice

of Constraint Programming (CP’98), Springer, Lecture Notes in Computer Science,

vol 1520, pp 417–431

Smith J (2002) Co-evolving memetic algorithms: Initial investigations. In: Parallel

Problem Solving from Nature PPSN VII, Springer Berlin / Heidelberg, Lecture

Notes in Computer Science, vol 2439, pp 537–546

Smith J (2008) Adaptive and Multilevel Metaheuristics, Springer, chap Self-Adaptation

in Evolutionary Algorithms for Combinatorial Optimisation, pp 31–57

54

Smith JE (2003) Protein structure prediction with co-evolving memetic algorithms. In:

Proc. Congress on Evolutionary Computation CEC ’03, vol 4, pp 2346–2353 Vol.4

Smith JE (2007) Co-evolving memetic algorithms: A review and progress report. IEEE

Transactions in Systems, Man and Cybernetics, part B 37(1):6–17

Smith JE, Fogarty TC (1997) Operator and parameter adaptation in genetic algo-

rithms. Soft Computing pp 81–87

Smith-Miles K (2008a) Towards insightful algorithm selection for optimisation using

meta-learning concepts. In: Proc. (IEEE World Congress on Computational Intelli-

gence). IEEE International Joint Conference on Neural Networks IJCNN 2008, pp

4118–4124

Smith-Miles KA (2008b) Cross-disciplinary perspectives on meta-learning for algorithm

selection. ACM Computing Surveys 41:6:1 – 6:25

Stephenson M, OReilly U, Martin M, Amarasinghe S (2003) Meta optimization: Im-

proving compiler heuristics with machine learning. In: Proceedings of the Conference

on Programming Language Design and Implementation (SIGPLAN03), San Diego,

CA, USA, pp 77–90

Storer RH, Wu SD, Vaccari R (1992) New search spaces for sequencing problems with

application to job shop scheduling. Management Science 38(10):1495–1509

Storer RH, Wu SD, Vaccari R (1995) Problem and heuristic space search strategies for

job shop scheduling. ORSA Journal of Computing 7(4):453–467

Sutton RS, Barto AG (1998) Reinforcement Learning: An Introduction. MIT Press

Tavares J, Machado P, Cardoso A, Pereira FB, Costa E (2004) On the evolution of

evolutionary algorithms. In: Keijzer M, O’Reilly UM, Lucas SM, Costa E, Soule T

(eds) LNCS 3003. Proceedings of the European Conference on Genetic Programming

(EUROGP’04), Coimbra, Portugal, pp 389–398

Tay JC, Ho NB (2008) Evolving dispatching rules using genetic programming for solv-

ing multi-objective flexible job-shop problems. Computers and Industrial Engineer-

ing 54(3):453–473

Terashima-Marın H, Ross P, Valenzuela-Rendon M (1999) Evolution of constraint sat-

isfaction strategies in examination timetabling. In: Genetic and Evolutionary Com-

putation COnference, GECCO’99, pp 635–642

Terashima-Marın H, Zarate CJF, Ross P, Valenzuela-Rendon M (2006) A ga-based

method to produce generalized hyper-heuristics for the 2d-regular cutting stock

problem. In: GECCO ’06: Proceedings of the 8th annual conference on Genetic and

evolutionary computation, ACM Press, New York, NY, USA, pp 591–598

Terashima-Marın H, Zarate CJF, Ross P, Valenzuela-Rendon M (2007) Comparing

two models to generate hyper-heuristics for the 2d-regular bin-packing problem. In:

Lipson H (ed) Genetic and Evolutionary Computation Conference, GECCO 2007,

ACM, pp 2182–2189

Terashima-Marın H, Ortiz-Bayliss JC, Ross P, Valenzuela-Rendon M (2008a) Hyper-

heuristics for the dynamic variable ordering in constraint satisfaction problems.

In: Ryan C, Keijzer M (eds) Genetic and Evolutionary Computation Conference,

GECCO 2008, ACM, pp 571–578

Terashima-Marın H, Ross P, Farias-Zarate CJ, Lopez-Camacho E, Valenzuela-Rendon

M (2008b) Generalized hyper-heuristics for solving regular and irregular bin packing

problems. Annals of Operations Research 1(1):1–10

Thabtah F, Cowling P (2008) Mining the data from a hyperheuristic approach us-

ing associative classification. Expert Systems with Applications: An International

Journal 34(2):1093–1101

55

Thompson J, Dowsland K (1996) General cooling schedules for a simulated annealing

based timetabling system. In: E K Burke PR (ed) Practice and Theory of Automated

Timetabling, Springer, Lecture Notes in Computer Science, vol 1153, pp 345–363

Tuson A, Ross P (1998) Adapting operator settings in genetic algorithms. Evolutionary

Computation 6:161–184

Vazquez-Rodrıguez JA, Petrovic S (2010) A new dispatching rule based genetic algo-

rithm for the multi-objective job shop problem. Journal of Heuristics (to appear)

Vazquez-Rodrıguez JA, Petrovic S, Salhi A (2007a) A combined meta-heuristic with

hyper-heuristic approach to the scheduling of the hybrid flow shop with sequence

dependent setup times and uniform machines. In: Baptiste P, Kendall G, Munier

A, Sourd F (eds) Proceedings of the 3rd Multidisciplinary International Scheduling

Conference: Theory and Applications (MISTA 2007)

Vazquez-Rodrıguez JA, Petrovic S, Salhi A (2007b) An investigation of hyper-heuristic

search spaces. In: Proceedings of the IEEE Congress on Evolutionary Computation

(CEC 2007), pp 3776–3783, DOI 10.1109/CEC.2007.4424962

Wah BW, Ieumwananonthachai A (1999) Teacher: A genetics-based system for learning

and for generalizing heuristics. In: Yao X (ed) Evolutionary Computation, World

Scientific Publishing Co. Pte. Ltd., pp 124–170

Wah BW, Ieumwananonthachai A, Chu LC, Aizawa A (1995) Genetics-based learning

of new heuristics: Rational scheduling of experiments and generalization. IEEE Trans

on Knowledge and Data Engineering 7(5):763–785

Wilson S (1995) Classifier systems based on accuracy. Evolutionary Computation

3(2):149–175

Wolpert D, Macready W (1997) No free lunch theorems for optimization. IEEE Trans

on Evolutionary Computation 1:67–82