Assessing the suitability of surrogate modelsin evolutionary optimization?

Martin Holena and Radim Demut

Institute of Computer Science, Academy of Sciences of the Czech RepublicPod Vodarenskou vezı 2, 18207 Prague

Abstract. The paper deals with the application of evolu-tionary algorithms to black-box optimization, frequently en-countered in biology, chemistry and engineering. In thoseareas, however, the evaluation of the black-box fitness is of-ten costly and time-consuming. Such a situation is usuallytackled by evaluating the original fitness only sometimes,and evaluating its appropriate response-surface model oth-erwise, called surrogate model of the fitness. Several kindsof models have been successful in surrogate modelling, anda variety of models of each kind can be obtained throughparametrization. Therefore, real-world applications of sur-rogate modelling entail the problem of assessing the suit-ability of different models for the optimization task beingsolved. The present paper attempts to systematically inves-tigate this problem. It surveys available methods to assessmodel suitability and reports the incorporation of severalsuch methods in our recently proposed approach to surro-gate modelling based on radial basis function networks. Inaddition to the commonly used global suitability of a model,it pays much attention also to its local suitability for a giveninput. Finally, it shows some results of testing several ofthe surveyed methods in two real-world applications.

1 Introduction

An important application area of evolutionary opti-mization algorithms [8, 31] is black-bock optimization,i.e., optimization of an objective function (in evolu-tionary terms called fitness) that cannot be describedexplicitly, but is known only from its evaluations ina finite number of points in the input space.Frequently, the fitness is evaluated in some empiricalway, through measurements or testing. This is typicalfor applications in biology, chemistry, or materials sci-ence [1]. In those domains, however, the fact that evo-lutionary algorithms rely solely on fitness evaluationscan be quite disadvantageous because the evaluation ofempirical functions encountered there is usually time-consuming and costly. For example in the evolution-ary optimization of catalytic materials [1, 12], wherea fitness describes the suitability of the material fora particular chemical reaction, its evaluation in onegeneration of the evolutionary algorithm needs sev-

? The research reported in this paper has been sup-ported by the Czech Science Foundation (GA CR) grantP202/11/1368.

eral days to several weeks of time and costs several tomany thousands of euros.

The usual way of dealing with a time-consumingand costly evaluation of an objective function is toevaluate such a function only sometimes, andevaluate its suitable response-surface model oth-erwise [19, 25]. In the context of evolutionary opti-mization, such a model is commonly called surrogatemodel of the fitness, and the approach is called sur-rogate modelling [10, 29, 32, 37] (occasionally also out-side that context [3, 23]). Because fitness is typicallyassumed to be highly nonlinear, nonlinear models areused as surrogate models. So far most frequently en-countered have been Gaussian processes [7, 28, 37](inspired by their success in response surface mod-elling [20, 21, 23, 35]), radial basis function (RBF) net-works [2, 37] and other kinds of feedforward neural net-works [16, 18].

Due to the applicability of different kinds of modelsto surrogate modelling, as well as due to the possibilityto construct a variety of models of each kind throughan appropriate parametrization, a large number of var-ious surrogate models can always be employed. There-fore, real-world applications of surrogate modelling en-tail the problem of assessing the suitability of differentmodels for the optimization task being solved. Unfor-tunately, no systematic attention seems to have beenpaid to that problem so far in the area of surrogatemodelling, the research in this area being focused onthe integration of surrogate models with evolutionaryoptimization algorithms, their adaptation to the opti-mization tasks, and on increasing the accuracy of theconstructed models [11, 14, 17, 26]. The present paperis an attempt to change the situation. We survey avail-able methods to assess model suitability, concentrat-ing in particular on local suitability of the model fora given input. Moreover, we give some results of test-ing several of the surveyed methods on two real-worldapplications of surrogate modelling.

In the following section, the principles of surrogatemodelling are recalled and their usefulness for evolu-tionary optimization is documented. The key sectionof the paper is Section 3, in which the most importantmethods for assessing model suitability are explainedand results of their testing are presented.

32 Martin Holena, Radim Demut

2 Surrogate modelling in evolutionaryoptimization

In evolutionary optimization, surrogate modelling isan approach in which the evaluation of the originalblack-box fitness is restricted to points considered to bemost important in the search for its global maximum,and its appropriate response-surface model is evalu-ated otherwise. Important for searching the global ma-ximum of a fitness function are on the one hand thehighest values found so far, on the other hand the di-versity of current population. Therefore, the selectionof points in which the original fitness is evaluated isalways based on some combination of those two crite-ria.

The different ways of interaction between the sys-tem input/output interface, the generated evolution-ary algorithm (EA) and the surrogate model can basi-cally be assigned to one of the following two strategies:

A. The individual-based strategy consists in choosingbetween the evaluation of the original fitness andthe evaluation of its surrogate model individual-wise, for example, in the following steps:

(i) An initial set E of individuals is collected inwhich the original fitness η was evaluated(e.g., individuals forming several first genera-tions of the EA).

(ii) The model is trained using pairs {(x, η(x)) :x ∈ E}.

(iii) The EA is run with the fitness η replacedby the model for one generation with a pop-ulation Q of size qP , where P is the desiredpopulation size for the optimization of η, andq is a prescribed ratio (e.g., q = 10 or q = 100).

(iv) A subset P ⊂ Q of size P is selected so as tocontain those individuals fromQ that are mostimportant according to the considered criteriafor the progress of optimization.

(v) For x ∈ P, the original fitness is evaluated.

(vi) The set E is replaced by E ∪ P and the algo-rithm returns to (ii).

B. The generation-based strategy consists in choosingbetween both kinds of evaluation generation-wise,for example, in the following steps:

(i) An initial set E of individuals in which theoriginal fitness η was evaluated is collected likein the individual-based strategy.

(ii) The model is trained using pairs {(x, η(x)) :x ∈ E}.

(iii) The EA is run with the fitness η replacedby the model for a number gm of generations,interactively obtained from the user, with pop-ulations P1, . . . ,Pgm of size P .

(iv) The EA is run with the original fitness η fora prescribed number ge of generations withpopulations Pgm+1, . . . ,Pgm+ge (frequently,ge = 1).

(v) The set E is replaced by E ∪ Pgm+1 ∪ · · · ∪Pgm+ge and the algorithm returns to (ii).

The fact that surrogate modeling is employed inthe context of costly or time-consuming objective func-tions effectively excludes the possibility to use thosefunctions for tuning surrogate modeling methods, andfor comparing different models and different ways oftheir combining with evolutionary optimization. Toget around this difficulty, artificial benchmark func-tions can be used, computed analytically but expectedto behave in evolutionary optimization similarly to theoriginal fitness. As an example, Fig. 1 shows the appli-cation of surrogate modelling to a benchmark functionproposed in [34] for the application area of optimiza-tion of catalytic materials (cf. [1]). The benchmarkfunction was optimized using the system GENACAT[13, 15], one of several evolutionary optimization sys-tems developed specifically for that application area.The evolutionary algorithm employed by GENACATis a genetic algorithm (GA) taking into account thecomposition and properties of catalytic materials. Assurrogate model, a RBF-network trained with datafrom all previous generations was used, combined withthe GA according to the individual-based strategy.The results shown in Fig. 1 clearly document that sur-rogate modelling substantially accelerates the searchfor the maximum of a fitness function.

3 Assessing the suitability of differentmodels

Instead of a single surrogate model, a whole set of mod-els F an be used. Then it is necessary to decide howsuitable each of them is to be evaluated instead of theoriginal fitness η. Typically, the suitability of a modelF ∈ F is assumed to be indirectly proportional tosome error ε(F ), defined on F and calculated usinga given sequence of data not used for the constructionof F . Consequently, the most suitable surrogate modelis the one fulfilling

F = arg minF∈F

ε(F ). (1)

There are various ways how ε(F ) takes into accountthe evaluation η(x) by the original fitness and the eval-uation F (x) by the model for given inputs x, e.g.,mean absolute error, mean squared error, root meansquare error, relative entropy, Kullback-Leibler diver-gence, . . . . There are also two basic ways how to assurethat the given sequence of data was not used for the

Assessing the suitability of surrogate models . . . 33

Fig. 1. Comparison of the highest values of the benchmark fitness function from [34] found by the evolutionary opti-mization system GENACAT [13, 15] without surrogate modeling and with an RBF network used as surrogate model,according to the individual-based strategy.

construction of F : single split and cross-validation, thelatter having the important advantage that all avail-able data are used both for model construction andfor the estimation of model error.

Let us exemplify the calculation of ε(F ) by recall-ing the definition of the root mean squared error onan input data sequence D:

RMSE(F ) = RMSED(F ) = (2)

=

√1

|D|∑x∈D

(F (x)− η(x))2,

where |D| denotes the cardinality of D. Thus the over-all root mean squared error of F based on a k-foldcrossvalidation with folds D1, . . . , Dk is:

RMSE(F ) =1

k

k∑i=1

RMSEDi(F ) = (3)

=1

k

k∑i=1

√1

|Di|∑x∈Di

(F (x)− η(x))2,

Observe that the most suitable model F in (1) de-pends on the considered set of surrogate models F ,but does not depend on the inputs in which it has tobe evaluated. Therefore, it can be called globally mostsuitable with respect to the given sequence of data. Itsobvious advantage is that it needs to be found onlyonce, and then it can be used for all evaluations, aslong as the set F does not change.

Global suitability of surrogate models based oncross-validation was tested in more than a dozen evo-lutionary optimization tasks. Here, we show results oftesting it in a task where F was a set of multilayer per-ceptrons (MLPs) with two hidden layers and differentarchitectures. They were restricted to have nI = 14input neurons, no = 3 output neurons, and the num-bers of hidden neurons nH1 in the first and nH2 inthe second layer fulfilling the heuristic pyramidal con-dition: the number of neurons in a subsequent layermust not exceed the number of neurons in a previouslayer. Consequently,

14 ≥ nH1 ≥ nH2 ≥ 3, (4)

34 Martin Holena, Radim Demut

Fig. 2. Comparison of the RMSE of 21 surrogate models on test data, i.e., data from the 7th generation of a geneticoptimization, with the RMSE-estimate obtained for those models by means of leave-one-out cross-validation on datafrom the 1st–6th generation.

which yields 78 different MLP architectures. They weretested as follows:

1. The employed GA was run for 6 generations usingthe original fitness.

2. For each of the considered 78 architectures, onesurrogate model was trained using all the availabledata from the 1st–6th generation.

3. The RMSE of each surrogate model on the datafrom the 1st–6th generation was estimated usingleave-one-out cross-validation, according to (3).

4. The 7th generation G7 of the genetic algorithmwas produced.

5. The models obtained in step 2 were used to predictthe fitness of x ∈ G7.

6. For x ∈ G7, also the original fitness was evaluated.7. From the results of steps 5–6, the RMSE of each

surrogate model on the data from the 7th genera-tion was calculated according to (2), with D = G7.

8. For each surrogate model, the RMSE calculated instep 7 was compared to the RMSE estimate fromstep 2.

Figure 2 visualizes the results of comparisons instep 8 for a subset of the considered surrogate models,

namely for the 21 MLP architectures that, in additionto (4), fulfil 6 ≤ nH1, nH2 ≤ 11. The visualized re-sults indicate that the rank of models according to theRMSE-based suitability estimated by means of leave-one-out cross-validation on the data from the 1st–6th

generation correlates with their rank according to theRMSE on the data from the 7th generation. We alsoquantified the extent of that correlation, using:

(i) Kendall’s rank correlation coefficient τ betweenthe ranks of models according to the suitabilitybased on RMSE and estimated using leave-one-out cross-validation, and according to the RMSEon test data from the 7th generation,

(ii) achieved significance level pτ of the test of rankindependence based on the correlation coefficient τobtained in (i).

The results were

τ = 0.77 and pτ = 1.7 · 10−8, (5)

which clearly confirm a strong correlation between therank of the model suitability and the rank of modelRMSE on test data.

Assessing the suitability of surrogate models . . . 35

3.1 Local suitability

Needless to say, the fact that a globally most suitablemodel achieves the least value of an error ε calculatedusing a given sequence of data does not at all meanthat it yields the most suitable prediction for every xin which fitness η can be evaluated. Therefore, we in-troduce the model Fx locally most suitable for x as

Fx = arg minF∈F

λ(F, x). (6)

Like ε in (1), λ in (6) denotes some error. Dif-ferently to ε, however, λ is defined on the cartesianproduct F × X , where X denotes the set of points inwhich η can be evaluated.

Recall that a surrogate model is evaluated in pointsx ∈ X in which η has not been evaluated yet. Hence,the calculation of the error λ(F, x) must not depend onthe value of η(x). Though in the context of surrogatemodelling, using such errors has not been reported yet,a number of error measures exist that could be usedto this end, most importantly:

– widths of confidence intervals [33];

– transductive confidence [9, 27, 36];

– estimation of prediction error relying on densityestimation [4];

– sensitivity analysis [5];

– several heuristics based on the nearest neighboursof the point of evaluation [4, 30];

– heuristic based on the variance of bagged mod-els [6];

We are currently extending the surrogate modelpresented in [2], which is based on RBF-networks, withthree of the above error measures:

(i) Width of prediction intervals for x ∈ X , i.e., ofconfidence intervals for Fx(x) in (6) based on thelinearization of the surrogate models and on theassumption of independent normally distributedresiduals. First, each F ∈ F is replaced with itsTaylor expansion of the 1st order, which is a lin-ear regression model FLIN with d + 1 parame-ters, where d is the dimensionality of points in X .Hence, F is replaced with

FLIN = {FLIN : F ∈ F}. (7)

Then for each considered x∈X , the element FMLEx

of FLIN corresponding to the maximum-likelihoodestimate of the d + 1 parameters is found usinga given training sequence of input-output pairs(x1, y1), . . . , (xp, yp). That allows to calculate foreach F ∈ F the error in (6) as

λ(i)(F, x) =

∣∣∣∣∣∣p∑j=1

(yj − FMLEx (xj))

2−

−(

1 +F1−α[1, p− d]

p− d

) p∑j=1

(yj − F (xj))2

∣∣∣∣∣∣ , (8)

where F1−α[1, p− d] denotes the 1− α quantile ofthe Fisher-Snedecor distribution with the degreesof freedom 1 and p− d.

(ii) Difference between the predicted value and the nea-rest-neighbours average is for k nearest neighboursxn1

, . . . , xnkcalculated according to

λ(ii)(F, x) =

∣∣∣∣∣∑kj=1 ynj

k− F (x)

∣∣∣∣∣ . (9)

(iii) Bagged variance requires to have some set B ofbasic models and to find, in B, a given numberm ofmodels bagged with respect to (x1, y1),. . . ,(xp, yp),i.e., globally most suitable with respect to boot-strap samples from (x1, y1), . . . , (xp, yp). Recallfrom (1) that to find such a bagged model, an er-ror ε(B) is needed, calculated using the respectivebootstrap sample. In our implementation, we al-ways use RMSE to this end. In its calculation ac-cording to (3), hence, D1, . . . , Dk are folds of thebootstrap sample. Using the bagged models, thefinal set of considered surrogate models is definedas

F =

F : F =

m∑j=1

BFj & BF1 , . . . , BFm ∈ B

,

(10)

and the bagged variance is calculated according to

λ(iii)(F, x) =1

m

m∑j=1

BFj (x)− 1

m

m∑j=1

BFj (x)

2

.

(11)

Because the extension of the surrogate modelfrom [2] with those three error measures has been im-plemented very recently, it is now in the course of test-ing in a second evolutionary optimization task. Here,a part of the results from the first task will be shown,concerning the optimization of catalytic materials forhigh-temperature synthesis of HCN [24]. In that task,F was a set of five RBF networks, each with a differentnumber of hidden neurons in the range 1–5. Those fivenetworks were tested in a similar way as was employedin the above MLP-case. In particular:

36 Martin Holena, Radim Demut

Fig. 3. Juxtaposition of the errors of predictions by RBF networks with 1–5 hidden neurons for 30 catalysts randomlyselected from the 7th generation of a genetic optimization of catalytic materials for high-temperature synthesis ofHCN [24], with the choices of the networks locally most suitable according to λ(i), λ(ii) and λ(iii), and with the globallymost suitable model.

1. The employed GA was again run for 6 generationsusing the original fitness.

2. For each of the 5 possible numbers of hidden neu-rons, one surrogate model was trained using all theavailable data from the 1st–6th generation.

3. The RMSE of each surrogate model on the datafrom the 1st–6th generation was estimated using10-fold cross-validation, according to (3).

4. Based on the result of step 3, the globally mostsuitable model was determined.

5. The 7th generation G7 of the genetic algorithmwas produced.

6. The models obtained in step 2 were used to predictthe fitness of x ∈ G7.

7. For F ∈ F and x ∈ G7, the errors λ(i)(F, x),λ(ii)(F, x), λ(iii)(F, x) were calculated.

Assessing the suitability of surrogate models . . . 37

8. The surrogate models locally most suitable forx ∈ G7 according to λ(i), λ(ii) and λ(iii) were de-termined.

9. For x ∈ G7, the original fitness was evaluated.10. For x ∈ G7, the absolute errors of the considered

surrogate models were calculated from the resultsof steps 6 and 9.

Figure 3 visualizes the locally most suitable modelsdetermined in step 8 for a subset of 30 randomly se-lected catalytic materials from the 7th generation. Thefact that for those catalysts the absolute errors of allfive RBF networks were calculated in step 10 allowsto juxtapose the choices of the locally most suitablemodels and the globally most suitable model (modelwith 2 hidden neurons) with the achieved absolute er-rors. The juxtaposition shows that the model with thelowest absolute error was nearly always assessed as thelocally most suitable by some of the three implementederror measures. Unfortunately, no one of them couldbe relied on in a majority of all cases.

4 Conclusion

This paper is, to our knowledge, a first attempt to sys-tematically investigate available methods for assessingthe suitability of surrogate models in evolutionary op-timization. In addition to the commonly used globalsuitability of a model, it paid much attention also toits local suitability for a given input. We have incorpo-rated three methods for assessing local suitability intoour recently proposed approach to surrogate modellingbased on RBF networks. The paper not only surveyedavailable methods for assessing suitability, but also de-scribed their testing and presented some of the testingresults.

The presented results clearly confirm the useful-ness of methods for assessing the suitability of surro-gate models: there is a strong correlation between theranks of models according to the RMSE-based globalsuitability estimated by means of leave-one-out cross-validation on data from the 1st–6th generation and ac-cording to the RMSE on the data from the 7th gen-eration, and for nearly every unseen input, the modelwith the lowest absolute error is indeed assessed asthe locally most suitable by some of the implementedmethods. Unfortunately, none of the methods is ableto correctly assess the locally most suitable model fora majority of the 7th generation, which shows that fur-ther research in this area is needed. We want to takeactive part in such research, pursuing the followingthree directions:(i) Implement and test further methods for assessing

local suitability, listed above in Subsection 3.1. Weconsider particularly interesting the transductive

confidence machine [9, 27, 36] because it is a novelmethod and has solid theoretical fundamentals.

(ii) Investigate whether the success of some of thetested methods for assessing local suitability de-pends on the kind of dataset (in terms of the num-ber of nominal attributes, number of continuousattributes, etc.) on which the surrogate modelshave been trained, or on their descriptive statis-tics. To this end, we want to make use of theGAME system [22], which collects a large amountof meta-data about the kind of the processed da-taset and about its descriptive statistics.

(iii) Modify and combine the tested methods, usingthe results obtained in (ii), to increase their suc-cess in assessing the local suitability of surrogatemodels.

References

1. M. Baerns and M. Holena: Combinatorial developmentof solid catalytic materials. Design of high-throughputexperiments, data analysis, data mining. Imperial Col-lege Press, London, 2009.

2. L. Bajer and M. Holena: Surrogate model for contin-uous and discrete genetic optimization based on RBFnetworks. In C. Fyfe, P. Tino, C. Garcia-Osorio, andH. Yin, (eds), Intelligent Data Engineering and Au-tomated Learning. Lecture Notes in Computer Sci-ence 6283, pp. 251–258. Springer Verlag, Berlin, 2010.

3. A.J. Booker, J. Dennis, P.D. Frank, D.B. Serafini, Tor-czon V., and M. Trosset: A rigorous framework foroptimization by surrogates. Structural and Multidisci-plinary Optimization, 17, 1999, 1–13.

4. Z. Bosnic and I. Kononenko: Comparison of ap-proaches for estimating reliability of individual regres-sion predictions. Data & Knowledge Engineering, 67,2008, 504–516.

5. Z. Bosnic and I. Kononenko: Estimation of individualprediction reliability using the local sensitivity analysis.Applied Intelligence, 29, 2008, 187–203.

6. L. Breiman: Bagging predictors. Machine Learning,24, 1996, 123–140.

7. D. Buche, N.N. Schraudolph, and P. Koumoutsakos:Accelerating evolutionary algorithms with gaussianprocess fitness function models. IEEE Transactions onSystems, Man, and Cybernetics, Part C: Applicationsand Reviews, 35, 2005, 183–194.

8. C.A.C. Coello, G.B. Lamont, and D.A. van Veld-huizen: Evolutionary algorithms for solving multi-objective problems, 2nd Edition. Springer Verlag,Berlin, 2007.

9. A. Gammerman, V. Vovk, and V. Vapnik: Learning bytransduction. In Uncertainty in Artificial Intelligence,pp.148–155. Morgan Kaufmann Publishers, San Fran-cisco, 1998.

10. D. Gorissen, I. Couckuyt, P. Demeester, T. Dhaene,and K. Crombecq: A surrogate modeling and adaptivesampling toolbox for computer based design. Journal ofMachine Learning Research, 11, 2010, 2051–2055.

38 Martin Holena, Radim Demut

11. D. Gorissen, T. Dhaene, and F. DeTurck: Evolution-ary model type selection for global surrogate modeling.Journal of Machine Learning Research, 10, 2009, 2039–2078.

12. M. Holena and M. Baerns: Computer-aided strategiesfor catalyst development. In G. Ertl, H. Knozinger,F. Schuth, and J. Weitkamp, (eds), Handbook of Het-erogeneous Catalysis, pp. 66–81. Wiley-VCH, Wein-heim, 2008.

13. M. Holena, T. Cukic, U. Rodemerck, and D. Linke:Optimization of catalysts using specific, descriptionbased genetic algorithms. Journal of Chemical Infor-mation and Modeling, 48, 2008, 274–282.

14. M. Holena, D. Linke, and U. Rodemerck: Evolu-tionary optimization of catalysts assisted by neural-network learning. In Simulated Evolution and Learn-ing. Lecture Notes in Computer Science 6457, pp.220–229. Springer Verlag, Berlin, 2010.

15. M. Holena, D. Linke, and U. Rodemerck: Generatorapproach to evolutionary optimization of catalysts andits integration with surrogate modeling. Catalysis To-day, 159, 2011, 84–95.

16. M. Holena, D. Linke, and N. Steinfeldt: Boosted neuralnetworks in evolutionary computation. In Neural In-formation Processing. Lecture Notes in Computer Sci-ence 5864, pp. 131–140. Springer Verlag, Berlin, 2009.

17. Y. Jin: A comprehensive survery of fitness approxima-tion in evolutionary computation. Soft Computing, 9,2005, 3–12.

18. Y. Jin, M. Husken, M. Olhofer, and B. Sendhoff: Neu-ral networks for fitness approximation in evolution-ary optimization. In Y. Jin, (ed.), Knowledge Incor-poration in Evolutionary Computation, pp. 281–306.Springer Verlag, Berlin, 2005.

19. D.R. Jones: A taxonomy of global optimization meth-ods based on response surfaces. Journal of Global Op-timization, 21, 2001, 345–383.

20. D.R. Jones, M. Schonlau, and W.J. Welch: Efficientglobal optimization of expensive black-box functions.Journal of Global Optimization, 13, 1998, 455–492.

21. J.P.C. Kleijnen, W. van Baers, and I. van Nieuwen-huyse: Constrained optimization in expensive simu-lation: Novel approach. European Journal of Opera-tional Research, 202, 2010, 164–174.

22. P. Kordık, J. Koutnık, J. Drchal, O. Kovarık,M. Cepek, and M. Snorek: Meta-learning approachto neural network optimization. Neural Networks, 23,2010, 568–582.

23. S.J. Leary, A. Bhaskar, and A.J. Keane: A derivativebased surrogate model for approximating and optimiz-ing the output of an expensive computer simulation.Journal of Global Optimization, 30, 2004, 39–58.

24. S. Mohmel, N. Steinfeldt, S. Endgelschalt, M. Holena,S. Kolf, U. Dingerdissen, D. Wolf, R. Weber, andM. Bewersdorf: New catalytic materials for thehigh-temperature synthesis of hydrocyanic acid frommethane and ammonia by high-throughput approach.Applied Catalysis A: General, 334, 2008, 73–83.

25. R.H. Myers, D.C. Montgomery, and C.M. Anderson-Cook: Response surface methodology: proces and prod-uct optimization using designed experiment. John Wi-ley and Sons, Hoboken, 2009.

26. Y.S. Ong, P.B. Nair, A.J. Keane, and K.W. Wong:Surrogate-assisted evolutionary optimization frame-works for high-fidelity engineering design problems.In Y. Jin, (ed.), Knowledge Incorporation in Evolu-tionary Computation, pp. 307–331. Springer Verlag,Berlin, 2005.

27. H. Papadopoulos, K. Poredrou, V. Vovk, and A. Gam-merman. Inductive confidence machies for regression.In Proceedings of the 13th European Conference onMachie Learning, pp. 345–356, 2002.

28. A. Ratle: Kriging as a surrogate fitness landscape inevolutionary optimization. Artificial Intelligence forEngineering Design, Analysis and Manufacturing, 15,2001, 37–49.

29. T. Ray and W. Smith: A surrogate assisted parallelmultiobjective evolutionary algorithm for robust engi-neering design. Engineering Optimization, 38, 2006,997–1011.

30. S. Schaal and C.G. Atkeson: Assessing the quality oflearned local models. In Advances in Neural Informa-tion Processing Systems 6, pp. 160–167. Morgan Kauf-mann Publishers, San Mateo, 1994.

31. R. Schaefer: Foundation of global genetic optimization.Springer Verlag, Berlin, 2007.

32. H. Ulmer, F. Streichert, and A. Zell: Model assistedevolution strategies. In Y. Jin, (ed.), Knowledge Incor-poration in Evolutionary Computation, pp. 333–355.Springer Verlag, Berlin, 2005.

33. E. Uusipaikka: Confidence intervals in generalized re-gression models. CRC Press, Boca Raton, 2009.

34. S. Valero, E. Argente, V. Botti, J.M. Serra, P. Serna,M. Moliner, and A. Corma: DoE framework for cat-alyst development based on soft computing techniques.Computers and Chemical Engineering, 33, 2009, 225–238.

35. J. Villemonteix, E. Vazquez, and E. Walter: Aninformational approach to the global optimization ofexpensive-to-evaluate functions. Journal of Global Op-timization, 44, 2009, 509–534.

36. V. Vovk, A. Gammerman, and G. Shafer: Algorithmiclearning in a random world. Springer Verlag, Berlin,2005.

37. Z.Z. Zhou, Y.S. Ong, P.B. Nair, A.J. Keane, and K.Y.Lum: Combining global and local surrogate models toaccellerate evolutionary optimization. IEEE Transac-tions on Systems, Man and Cybernetics. Part C: Ap-plications and Reviews, 37, 2007, 66–76.