-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
1/15
Sensitivity analysis in discrete-event simulationusing fractional factorial designsJAB Montevechi*, RG de Almeida Filho, AP Paiva, RFS Costa and AL Medeiros
Instituto de Engenharia de Producao e Gestao, Universidade Federal de Itajuba, Itajuba, Brasil
This paper presents a sensitivity analysis of discrete-event simulation models based on a twofold approach formed by
Design of Experiments (DOE) factorial designs and simulation routines. This sensitivity analysis aim is to reduce the
number of factors used as optimization input via simulation. The advantage of reducing the input factors is that
optimum search can become very time-consuming as the number of factors increases. Two cases were used to illustrate
the proposal: the first one, formed only by discrete variable, and the second presenting both discrete and continuous
variables. The paper also shows the use of the Johnsons transformation to experiments with non-normal response
variables. The specific case of the sensitivity analysis with a Poisson distribution response was studied. Generally,
discrete probability distributions lead to violation of constant variance assumption, which is an important principle in
DOE. Finally, a comparison between optimization conducted without planning and optimization based on sensitivity
analysis results was carried out. The main conclusion of this work is that it is possible to reduce the number of runs
needed to find optimum values, while generating a system knowledge capable to improve resource allocation.
Journal of Simulation advance online publication, 13 November 2009; doi:10.1057/jos.2009.23
Keywords: discrete-event simulation; design of experiments; optimization
1. Introduction
Manufacturing system simulation modelling dates back to at
least the early 1960s (Law and Mccomas, 1998) and is one of
the most popular and powerful tools employed to analyze
complex manufacturing systems (OKaneet al, 2000, Banks
et al, 2005). According to OKane et al(2000), one way toforecast the behaviour of these systems is the use of discrete-
event simulation, which consists of modelling a system where
changes occur at discrete-time intervals. This is appropriate
for manufacturing systems as their behaviour changes in
such a way.
Some manufacturing issues addressed by simulation
include specifying the need and quantity of equipment and
personnel, performance evaluation and evaluation of opera-
tional procedures (Law and Mccomas, 1998). The objectives
of simulation are classified as performance analysis,
capacity/constraint analysis, configuration comparison, op-
timization, sensitivity analysis and visualization (Harrellet al, 2000).
The optimization via simulation deserves special attention.
Harrell et al (2000) define optimization as the process of
trying different combinations of values for the variables that
can be controlled in order to seek for the combination of
values that provides the most desirable output from the
simulation model. However, as the number of variables
increases, the optimization phase becomes more time-
consuming.
To avoid this drawback, sensitivity analysis can be used to
select the variables that are responsible for most of the
variation in the experimental response, eliminating from the
model those variables that are not statistically significant. In
the simulation, sensitivity analysis can be interpreted as a
systematic investigation of simulation outputs according to
the input parameters chosen from the model (Kleijnen,
1998).
In this paper, factorial designs are used to perform a
sensitivity analysis of a simulation model. First, a fractional
factorial design is used to determine the no statistically
significant parameters. Once they are determined, a 2k full
factorial design can be built with the remainders. The
objective is to show how factorial designs can speed up the
simulation optimization to reach the best solution. Two
comparative studies between an all model factors optimiza-tion and another optimization using only the statistically
significant model factors were carried out.
The remainder of this paper is organized as follows: the
next section introduces discrete-event simulation and pre-
sents considerations on performing a simulation. After that,
fractional factorial designs are introduced. And finally, the
considerations on the construction of two simulation
models, the experiments and results analysis are presented.
The paper concludes with some considerations on the
integration between factorial designs and optimization.
*Correspondence: JAB Montevechi, Universidade Federal de ItajubaIEPG, Av. BPS, 1303Caixa Postal: 50, Itajuba, MG, 37500-903,Brasil.E-mail: [email protected]
Journal of Simulation (2009), 115 r 2009 Operational Research Society Ltd. All rights reserved. 1747-7778/09
www.palgrave-journals.com/jos/
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
2/15
2. Discrete-event simulation
Simulation is the process of designing a model of a real
system and conducting experiments with such a model. This
is done with the purpose of understanding the behaviour
of the system and/or evaluating various strategies for the
operation of the system (Shannon, 1998). Some advantages
of simulation are:
One can simulate systems that already exist as well as
those that are capable of being brought into existence.
Simulation allows one to identify bottlenecks in informa-
tion, material and product flows and to test options for
increasing flow rates.
It allows one to gain insights into how a modelled system
actually works and to understand which variables are
most important to performance.
A significant advantage of simulation is its ability to let
one experiment with new and unfamiliar situations and to
answer what if questions.
Some simulation disadvantages are: model building
requires special training, simulation results can be difficult
to interpret, simulation modelling and analysis can be time
consuming and expensive, the misuse of simulation to solve
problems instead of analytical solution when it is possible or
even preferable, and each run of a stochastic simulation
model produces only estimates of the true characteristics of
model for a particular set of input parameters (Law and
Kelton, 2000; Banks et al, 2005).
In this study, a discrete-event simulation is used. It
consists of modelling a system as it evolves over time by a
representation in which the state variables change instanta-neously at separate points in time (Law and Kelton, 2000).
This methodology is ideal to be applied to manufacturing
systems because they exhibit discrete production changes
(OKane et al, 2000).
According to Strack (1984), Law and Kelton (2000) and
OKaneet al(2000), some characteristics found on problems
to be analyzed that justified the use of simulation are:
real-world systems with stochastic elements cannot be
accurately described by a mathematical model that can
be evaluated analytically;
it is easier to obtain results from a simulation model thanusing an analytical method;
experimentation is impossible or very difficult in a real
world system; and
the need of long-period time studies or alternatives that
physical models do not provide.
3. Optimization via simulation
Optimization via simulation is the process of trying different
combinations of values for variables that can be controlled
in order to seek for the combination of values that provides
the most desirable output from the simulation model
(Harrell et al, 2000). According to Fu (2002), the integration
between optimization and simulation is recent and has been
occurring since the end of the last millennium; the relation-
ship commonly encountered in commercial software is a
subservient one where the optimization routine is an add-on
to the simulation engine. This optimization routine needs the
simulation engine outputs to find the parameter set which
leads to the best solution.
There are several techniques for optimization. Some of
them are based on heuristics. According to Silva and
Montevechi (2004), heuristic techniques accomplish good
solutions and eventually find the optimum solution. How-
ever, it is not possible to state that the solutions found
by these techniques are the best ones, because heuristics do
not test all possible responses.
4. Fractional factorial design
According to Montgomery (2001), when an experiment
involves the study of two or more factors, the most efficient
strategy is the factorial design. In this strategy, the factors
are altered together not one at a time, it means that for each
complete run or replica all possible combination levels are
investigated (Montgomery and Runger, 2003).
When the interest is to study the effects of two or more
factors, factorial designs are more efficient than the one-
factor-at-a-time approach (Montgomery and Runger, 2003).
By a factorial design, it means that in each complete
trial or replication of the experiment all possible combina-
tions of the levels of the factors are investigated. For
example, if there arealevels of factor A andblevels of factor
B, each replicate contains all ab treatment combinations
(Montgomery, 2001). Factorial designs are the only way
to find factor interactions (Montgomery, 2001; Montgomery
and Runger, 2003) avoiding incorrect conclusions when
factors interactions are presented. The factorial design major
issue is the exponential growth of the level combination as
the number of factors increases (Kleijnen, 1998).
The fractional factorial design deals with this by selecting
a subset of all possible points of a factorial design and the
simulation is run only for these points (Law and Kelton,
2000). Therefore, the fractional factorial design providesan alternative to obtain good estimates of main effects and
low order interactions but requires a fraction of computa-
tional work of a full factorial design (Law and Kelton, 2000).
5. System modelling
To accomplish the objective of this paper, two manufactur-
ing cells from different companies have been modelled. The
first one is an application in a Brazilian weapon factory
2 Journal of Simulation Vol.] ], No. ] ]
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
3/15
whereas the second is an application in a multinational
automotive plant.
5.1. Brazilian weapon factory
The objective of the factory is to increase the throughput by
adding new equipment to the cell. This model is determinis-tic. It means that no source of randomness is considered and
all input data are kept constant (Law and Kelton, 2000;
Banks et al, 2005).
The following information was gathered about the cell:
nine locations;
number of workers available (14 resources); and
the process and how long they last (37 activities), shifts
and part routes.
Promodel 7.0 (http://www.promodel.com/products/
promodel/), which is a discrete-event simulator for manu-
facturing and material handling, was used to build the
models. Figure 1 shows a screen of the models animation in
Promodel 7.0s.
The model verification and validation were performed in
two different ways. Initially, a plant expert analyzed whether
the behaviour of the model would seem reasonable
(Kleijnen, 1995; Sargent, 1998). Then, real historical data
were compared with simulated results.
In order to accomplish a comparison study, an investment
scenario was created. In this scenario, a loan was taken
in order to buy new equipment. The loan payment is made
on a monthly basis and this payment is subtracted from
monthly profit, which is determined by the monthly
production multiplied by unitary profit. The problem is to
select the optimum set of equipment for which the increase
of profit compensates the additional purchase cost of the
equipment. There are nine simulation model parameters
which can be changed. They represent how much equipment
is available to perform a specific task or operation. They
can assume two possible values: 1 or 2. Table 1 presents
these parameters. In this case, the Design of Experiments
(DOE) factors are represented by discrete variables.
The optimum set of equipment is determined by three
approaches. The first one performs several experiments
to identify the main factors and to get a mathematical
model that will be the objective function in an optimization
with Solvers (http://www.solver.com/). After identifying
the statistically significant simulation factors by using a two-
sample t hypothesis test (an usual procedure from any
statistic package), the original fractional factorial design can
be converted into a full factorial, eliminating from the design
Figure 1 Screen of the Brazilian weapon factory models animation.
Table 1 Variable factor assignment for the Brazilian weaponfactory application
Variable Factor Low level ()
High level( )
A Quantity of machines type 1 1 2B Quantity of machines type 2 1 2C Quantity of machines type 3 1 2D Quantity of machines type 4 1 2E Quantity of machines type 5 1 2F Quantity of machines type 6 1 2G Quantity of machines type 7 1 2H Quantity of machines type 8 1 2J Quantity of machines type 9 1 2
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 3
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
4/15
the no statistically significant terms. As these parameters are
also important to the simulation arrangement, despite not
being statistically significant, they can be kept constant in
proper levels. Comparatively, a second approach can be
established by using the main factors identified at the
experiments DOE as input for the optimization via
Simrunners. The optimization software used was Simrunner
3.0s, which uses Genetic Algorithms and is packaged along
with Promodel 7.0s.
Finally, the third approach is performed using all nine
factors in optimization via Simrunners.
5.2. Multinational automotive plant
The objective of this plant is to reduce the production lead
time by adding new operators and reducing setup times.
Production lead time means the time that one batch takes
from the moment raw materials are received to when the
finished products exit. This is a random model, that is, it is
governed by random variables.
The following information was gathered about the cell:
22 locations;
number of workers (six resources); and
the process and their durations (32 activities), shifts and
part routes.
Figure 2 shows a screen of the models animation in
Promodel 7.0.
Similarly to the first application, this model was verified
and validated through process experts analysis and compar-
ison between real historical data and simulated results.
The problem is to minimize the production lead time.
There are 10 simulation model parameters which can be
changed. They represent the number of operators that are
available to perform the cell activities and the setup mean
time of four machines (total setup and partial setup). Table 2
presents these parameters. It can be observed that in this
case, the factors related to the simulation are represented by
both continuous and discrete variables. For sake of
simplicity, the standard deviation of each machine setup
time was kept constant.
The optimum set of parameters is determined by three
approaches similar to the first application.
6. Experimentation
Initially, the experiments are planned to identify the most
statistically significant model factors. After that, the experi-
ments generate an objective function that is used in the
optimization via Solver. Then, these most statistically
significant factors are used as input data for optimization
via Simrunner. Also, this model is optimized using all factors
as input data in order to compare the results.
Figure 2 Screen of the multinational automotive plant models animation.
4 Journal of Simulation Vol.] ], No. ] ]
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
5/15
6.1. Identifying important factors
According to Law and Kelton (2000), in simulation,
experimental designs provide a way to decide which specific
configurations to simulate before the runs are performed so
the desired information can be obtained with the lowest runs
of simulation. For instance, considering the second applica-
tion where there are 10 factors, if a full factorial experiment
were chosen, it would be necessary 210 1024 runs. There-
fore, a screening experiment must be considered.
Screening or characterization experiments are experiments
in which many factors are considered and the objective is to
identify those factors (if any) that have large effects
(Montgomery, 2001). Typically, screening experiment
involves using fractional factorial designs and it is performed
in the early stages of the project when many factors are likely
considered to have little or no effect on the response
(Montgomery, 2001). According to this author, in this
situation it is usually best to keep the number of factorslevels low.
6.2. Brazilian weapon plant
The experimental design adopted here was a two-level nine-
factor fractional factorial with resolution IV. Resolution IV
means no main effect is aliased with any other main effect or
with any two-factor interaction, but two-factor interactions
are aliased with each other (Montgomery, 2001). The
available resolution IV designs are 2IV93 64 runs and
2IV94 32 runs.
The 2IV94 design was chosen because it has fewer runsthan 2IV
93 design. If necessary, the former design collapses
into a fiveor lessfactor full factorial. As preliminary
studies have shown that five factors (C, D, E, G and H) are
critical to cell performance, this design works well. The
generators for this design are F7BCDE, G7ACDE,
H7ABDE and J7ABCE, and the defining relations
are IBCDEFACDEGABDEHABCEJABFG
ACFH ADFJBCGH BDGJCDHJ DEFGH
CEFGJBEFHJAEGHJABCDFGHJ. Table 1 shows
the factor assignment to the variables of the design.
Table 3 shows the design matrix for principal fraction with
the results obtained for each run. Analyzing the response
variable (Profit) contained in this table, it can be observed
that this response variable is not normal. The aforemen-
tioned response (Profit) was determined multiplying the
monthly production by a unitary profit. Mathematically, theresulting output variable can be interpreted as a number of
units produced in a month, which is typically a Poisson
distribution case. According to many researchers (Bisgaard
and Fuller, 1994; Lewis et al, 2000; Montgomery, 2001)
when using counts as the experimental response, the
assumption of constant variance made with all standard
analysis is violated. Then, a common method for dealing
with this problem is to transform the data before the
significance analysis, so that the assumption of constant
variance is more probable. To verify if the experimental data
really follows a Poisson distribution as discussed early, a
Goodness-of-fit test based on the chi-square distribution can
be applied (Haldar and Mahadevan, 2000). In this test, the
null hypothesis (H0) indicates the observed data follows a
Poisson distribution. To obtain the chi-square statistic it is
necessary to calculate the probability density function for
Poisson distribution, which can be written as Equation (1):
PX ni ellni
ni! X 0; 1; 2;4 1
To accomplish with the aims of the chi-square test, the
expected frequency Eiof the data, can be calculated as:
Ei N ellni
ni!
2
In Equation (2), Nis the total number of observations inthe sample (N 32 in this case), l is the mean of Poisson
distribution andniis the number of observation counted for
each class.
For the specific case of the Goodness-of-fit test, where there
is only one column to the data, the chi-square statistic becomes
w2TXri1
ni Ei2
Ei4w2m1k;1a 3
In the Equation (3), ni is the frequency observed in the
sample, m is the used number of classes used in distribution
Table 2 Variable factor assignment for the multinational automotive plant application
Variable Factor Low level () High level( )
A Quantity of operators 2 4B Mean time of the processing of the operators 2 1C Mean time of the total setup to machine_01 109 83D Mean time of the partial setup to machine_01 39 21
E Mean time of the total setup to machine_02 154 105F Mean time of the partial setup to machine_02 80 55G Mean time of the total setup to machine_03 152 115H Mean time of the partial setup to machine_03 55 35J Mean time of the total setup to machine_04 74 46K Mean time of the partial setup to machine_04 49 28
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 5
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
6/15
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
7/15
significance of the main and the interaction effects using the
conventional bilateral t-test or ANOVA. The standard
analysis procedure for an unreplicated two-level design is
the normal plot of the estimated factor effects. However,
these designs are so widely used in practice that many formal
analysis procedures have been proposed to overcome the
subjectivity of the normal probability (Montgomery, 2001).
Ye and Hamada (2001), for instance, recommend the use of
Lenths method, a graphical approach based on a Pareto
Chart for the error term. If the error term has one or more
degrees of freedom, the line on the graph is drawn at t, where
t is the (1a/2) quantile of t-distribution with degrees of
freedom equal to the (number of effects/3) (as shown in
Figure 4). The vertical line in the Pareto Chart is the margin
of error, defined as ME tPSE. Lenths pseudo standard
error (PSE) is based on sparsity of the effects principle,
which assumes the variation in the smallest effects is due to
the random error. To calculate PSE the following steps are
necessary: (a) calculates the absolute value of the effects; (b)
calculates S, which is 1.5median of the step (c); calculatesthe median of the effects that are less than 2.5 S and (d)
calculates PSE, which is 1.5 median calculated in step (c).
Examining the Pareto Chart in Figure 4, it can be noted
that at a significance level of 10% only factor C (Quantity of
equipment 3) and the interactions BC and CE are significant.
According to Montgomery (2001), if the experimenter can
reasonably assume that certain high-order interactions are
negligible, information on the main effects and low-order
interactions may be obtained. Otherwise, when there are
several variables, the system or process is likely to be driven
primarily by some of the main effects and low-order
interactions. Therefore, taking into consideration that
factors A, D, F, G, H and J are not significant, though
they are necessary to the simulation process, in the next step
of the experimentation process these factors must be kept at
low level (1), once these levels present higher profit values.
The factors B and E were kept in the set of meaningful input
variables because they are placed in the interactions BC and
CE, respectively. The main effects for transformed response
are shown in Figure 5. It is noticed that factors B (Quantity
of equipment 2) and C (Quantity of equipment 3) have
positive effect. These factors increase profit when they are
increased (high level). The other factors have negative effects
that decrease profit when they are increased (high level). The
analysis of the interactions in a fractional factorial is not
recommended as the confounding among the effects are
tremendous. However, in a strict sense, it is possible to use
the aliases (confounding) among the main and interaction
effects to explain why only the factors B, C and E were
chosen to compose the new design. Examining the con-founding among the statistically significant terms C, BC and
CE and the other factors and interactions of the 2IV94
fractional factorial design, it is possible to notice that the
factor C is aliased with a three-factor interactions AFH,
BGH and DHJ, which may be neglected according to the
resolution of the chosen design and sparsity of the effects
principle (Montgomery, 2001). The two-way interaction CE
is aliased with three-way interactions ABJ, ADG, BDF and
FGJ, which may also be discarded. The interaction BC is
aliased with the two-way interaction GH and the three-way
Percent
400000300000200000
99
90
50
10
1
N 32
AD 2,809
P-Value
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
8/15
interactions AEJ and DEF. Neglecting these three-way
interactions and considering that GH is formed by two weak
effects (G and H), besides being aliased with three-way
interactions, there is no reason to believe that any of these
factors is statistically significant. The alias structure used in
this analysis is available in many statistics packages.
As only factors B, C and E have presented some evidence
of significance, the 2IV94 design can be converted into a
replicated 23 full factorial design. Figures 6 and 7 show the
analysis for this new design, where it is possible to notice
that factors B and C must be changed to the high level (2) to
maximize the transformed response and, consequently, the
Term
Effect
ABBDAGAFEHAD
EEGAEEJEFDE
AEFCJACAHCDBH
HB
AJGDFJA
BECEBC
C
1.41.21.00.80.60.40.20.0
0.497
Lenth's PSE = 0.275013
Figure 4 Pareto Chart for 2IV94 Fractional design with significance level a 10%.
MeanofTransformedresponse
0.5
0.0
-0.5
1-1 1-1 1-1
0.5
0.0
-0.5
1-1 1-1 1-1
0.5
0.0
-0.5
1-1 1-1 1-1
A B C
D E F
G H J
Figure 5 Main effects plot for transformed Profit.
8 Journal of Simulation Vol.] ], No. ] ]
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
9/15
process Profit. Although the main effect E is not statistically
significant, observing the interaction plot of Figure 7, it is
clear that choosing factor E placed in the high level
combined with factors B and C in their respective higher
levels, the response increase considerably. However, in order
to check if it is the best solution, an optimization using these
three factors is performed. Also, an optimization using all
nine factors is performed to compare their performances.
6.3. Multinational automotive plant
The experimental design adopted here was a two-level
10-factor fractional factorial with resolution IV. The avail-
able resolution IV designs are 2IV93 64 runs and 2IV
94 32
runs. The 2IV93 design was chosen because it has a higher
number of experiments (Montgomery, 2001). Table 2 shows
the factor assignment to the variables of the design.
Term
Standardized Effect
E
B
BCE
BE
CE
BC
C
6543210
1,711
Figure 6 Pareto Chart for full factorial design with significance levela 10%.
MeanofJohnson
0.5
0.0
-0.51-1 1-1
1-1
0.5
0.0
-0.5
B C
E
B
C
E
1-1 1-1
1
0
-1
1
0
-1
B-11
C-11
Main Effects Plot (data means) for Johnson Interaction Plot (data means) for Johnson
Figure 7 Factorial and interaction plots for 2
3
full factorial design.
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 9
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
10/15
The design matrix for main fraction with the results
obtained was omitted because it is not the centre of interest
of this paper. Analyzing the response variable (production
lead time), it can be verified through a normality test that
this response variable is normal.
Similar to the first application, the 2IV93 fractional factorial
design used here is unreplicated. Examining the Pareto Chart
in Figure 8, it can be noted that at a significance level of 5%
only factors A (number of operators), E (mean time of the
total setup for machine 2) and the interactions AC, FH and
EG are significant. Therefore, taking into consideration that
factors B, C, D, G, J and K are not significant, though they
are necessary to the simulation process, in the next step of
the experimentation process these factors must be kept at
low level. Factors F and H were kept in the set of statistically
significant input variables because they are placed in the
interactions FH.
The aliases (confounding) among the main and interaction
effects is used to explain why only factors A, E, F and H
were chosen to compose the new design. Examining the
confounding among the statistically significant terms A, E,AC, FH and EG and the other factors and interactions of
the 2IV93 fractional factorial design, it is possible to notice that
factor C is aliased with a three-factor interaction BGH and
with a four-factor interactions ABEK, ADFH, BDFG and
EGHK, which may be neglected according to the resolution
of the chosen design and sparsity of the effects principle
(Montgomery, 2001). Similarly, factor E is aliased with a
four-factor interaction. The two-way interaction AC is
aliased with three-way interactions BEK and DFH and
with four-factor interactions BCGH and EFGJ, which may
also be discarded. The interaction EG is aliased with the
three-way interactions CHK and DHJ and with the four-
factor interactions ACFJ and ADFK, neglecting these three-
way and four-way interactions. Similarly, the two-way
interaction FH is aliased with a three-way and a four-factor
interaction.
The main effects for production lead time are shown in
Figure 9. It is noticed that factor A (number of operators)
has a strong negative effect on production lead time. This
factor decreases the production lead time when it is increased
(high level). Factor E (mean time of the total machine 2
setup) also reduces the production lead time when it is
increased (high level).
As only factors A, E, F and H have presented some
evidence of significance, the 2IV93 design can be converted in a
replicated 24 full factorial design. Figures 10 and 11 show the
analysis for this new design, where it is possible to notice
that factors A, E, F and H must be changed to the high level
to minimize the lead time.
This finding can be explained in practice as follows: the
company needs to reduce production lead time to increaseoutput and to answer to its customers, then it is possible to
change 10 factors to reach this objective, however, only four
factors are significant to reduce production lead time. So, the
company can apply its efforts only by adding new operators
and by reducing three setup times of two different machines
(2 and 3) that it will reach its objective.
A mathematical model can be obtained from the full
factorial analysis, as is shown by Equation (1), where the
characters represent the coded values of the respective
factors. A minimum production lead time can be calculated
Term
Effect
ADGFJ
CEAEH
BCAFDGHJBK
DAEF
EKEH
AGJAH
ACJCFBF
AGKBJ
BFGEJH
ADEGFH
EAC
A
1.81.61.41.21.00.80.60.40.20.0
0.279
Factor
EF
GH
J
Name
TS_04K
ABCD
Lenth's PSE = 0.134063
num_operpro_timeTS_01PS_01
TS_02PS_02
TS_03PS_03
PS_04
Figure 8 Pareto Chart for 2IV104 fractional design with significance level a 5%.
10 Journal of Simulation Vol. ] ], No. ] ]
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
11/15
by using linear programming via Solver from Excel,
considering Equation (4) as the objective function and the
values shown in Table 2 for the restrictions of the statistically
significant factors (A, E, F and H). This procedure points
out the minimum production lead time as 8.59 h. The values
to variables A, E, F and H are shown in Table 5.
Leadtime 9:8656 0:8325A
0:1722E 0:0087F
0:1278G 0:1478FG
4
In order to make a comparison between the current
approach and the genetic algorithms (GAs) approach, an
MeanofProductionLead
Time
10.4
9.6
8.8
42 12 83109 2139
10.4
9.6
8.8
105154 5580 115152 3555
10.4
9.6
8.84674 2849
num_oper pro_time TS_01 PS_01
TS_02 PS_02 TS_03 PS_03
TS_04 PS_04
Figure 9 Main effects plot for lead time.
Term
Standardized Effect
BCD
BC
ACD
AB
C
ABCD
ABD
AC
ABC
BD
AD
D
CD
B
A
121086420
2.01
Factor
PS_03
Name
ABCD
num_operTS_02PS_02
Figure 10 Pareto Chart for full factorial design with significance levela 5%.
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 11
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
12/15
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
13/15
6.5. Optimization using all factors
Brazilian weapon factory. In this optimization phase, the
nine parameters presented in Table 1 are selected as inputs.
The values these parameters can assume are 1 or 2. The
objective function is to maximize profit, as presented
earlier.
After 98 runs, the Simrunner stops the search. The bestresult found is 411.327, which corresponds to experiment 55
as presented at Figure 13 and the correspondent parameter
values are presented in Table 7.
Multinational automotive plant. The 10 parameters
presented in Table 2 are selected as inputs. The values
these parameters assumed were also shown in Table 2. The
objective function is to minimize the production lead time,
as presented earlier.
After 2571 runs (3 h 20 min), the Simrunner stops the
search. The best result found is 7.42 h and the correspondent
parameter values are presented in Table 8.
6.6. Result analysis
Brazilian weapon factory. Table 9 shows the results
obtained by the three procedures. The three procedures
lead to the same results indicating that there is coherency
among them. Taking into consideration the number of runs
necessary to optimize the model, it is clear the advantage of
determining previously the main parameters and then
proceeding the optimization using them instead of proceed-
ing with the optimization using all factors. The former
demanded 32 8 40 runs to obtain the best result against
the 98 runs from the latter, a reduction of 59% in the
number of runs. Considering only the runs used in the
Factorial Design, 32 runs versus 98 runs from optimization
using all factors, the reduction is about 67%.
For this application, since the optimization factor
levels and the Factorial Design levels are equal, the
optimization using three factors seems meaningless.
However, other applications where the optimization factors
could have several levels or even were represented by
continuous variables, the Factorial Designs would only
identify the main factors without specifying their optimum
values.
Table 7 Best solution for optimization using all factors for the
Brazilian weapon factory application
Parameter Value
Neq_Op050 1Neq_Op052 1Neq_Op070 1Neq_Op080 2Neq_Op082 2Neq_Op100 1Neq_Op110 1Neq_Op120 2Neq_Op170 1
Table 8 Best solution for optimization using all factors for themultinational automotive plant application
Parameter Value
Quantity of operators 4Processing time of the operators N(1; 0.30)Total setup time of the machine_01 N(102; 42)Partial setup time of the machine_01 N(21; 12)Total setup time of the machine_02 N(126; 22)Partial setup time of the machine_02 N(55; 46)Total setup time of the machine_03 N(119; 41)
Partial setup time of the machine_03 N(35; 4)Total setup time of the machine_04 N(54; 33)Partial setup time of the machine_04 N(33; 15)
Table 9 Results from the three procedures for the Brazilianweapon factory application
Parameter Design of experiments
Optimizationusing three
factors
Optimizationusing allfactors
A 1 1 (*) 1
B 1 1 (*) 1C 1 1 (*) 1D 2 2 2E 2 2 2F 1 1 (*) 1G 1 1 (*) 1H 2 2 2J 1 1 (*) 1Result (Profit) 411 327 411 327 411 327
Number of runs 32 8 98
Note: The parameters identified with (*) were not used as input foroptimization. They were kept at low level (1).
Figure 13 Performance measures plot for optimization usingall factors.
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 13
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
14/15
Multinational automotive plant. Table 10 summarizes
the results obtained by the three procedures. The optimiza-
tion with all factors generates the lowest production lead
time and spent 3.2 h of optimization, whereas the optimiza-
tion with only the statistically significant factors lead to a
value to the production lead time 12% higher (52 min), by
spending 0.3 h in optimization plus 3 h in experimentation
through factorial designs. At last, the optimization through
the mathematical model obtained from the factorial design
and solved by Solver lead to a value to the production
lead time 16% higher (70 min) than the first procedure
(using all factors). It took 0.2 h in optimization plus 3 h in
experimentation through factorial design. Taking into
account the objective of the analysis, that is, to find the
best production lead time, the best procedure is
the optimization with all the factors. By doing it this way,
the computational time spent is practically the same in
comparison to the other procedures.
Additionally, the human effort spent in the optimization
with all factors is the lowest when compared to building 64
experiments (simulation models) in the factorial designs.
However, in practice, to get the results indicated by theoptimization with all factors, it would be necessary to
modify eight setup times, one processing time and the
number of operators. It means that the company would
spend more money and time by changing all these factors
than by changing only the four statistically significant
factors, that is the result of the factorial designs (three setup
times and the number of operators). Thus, the factorial
designs can be used to point out the main parameters
and then the optimization can be used to specify their
optimum values.
7. Conclusions
The objective of this work was to show how a sensitivity
analysis using Factorial Designs can help the simulation
optimization to reach the best solution. Initially, to
accomplish this objective a fractional factorial design was
used to identify the most statistically significant effects of
two models. As the experimental response variable of thefirst application does not follow a normal distribution, it was
necessary to apply a transformation to make it normal. As
the response variable follows a Poison distribution, the
Johnson transformation was used. The analysis was
performed using the Lenths method because the fractional
factorial design was unreplicated which made the bilateral
t-test or ANOVA not possible to assess the significance
of the main and interactions effects.
Afterwards, three optimization trials/tests were carried
out: two using the factors previously identified, and the other
using all factors of the model. For these applications, it is
clear the advantage of determining previously the mainfactors and then proceeding the optimization using them
(Solver or Simrunner) instead of proceeding with the
optimization using all factors. Beyond this discussion, it
must be pointed out that the DOE approach improves the
manufacturing system understanding, generating further
knowledge about the importance and significance of each
resource used, which in turn favors the improvement of the
decision-making process. More than how many resources
are needed to optimize a system, improving its productivity,
increasing its profits and reducing costs, despite of the
relative time spent in the construction of the models,
this twofold approach (DOE/Simulation) elucidates how
the resource can be efficiently changed and employed.
AcknowledgementsThe authors acknowledge CNPq, PadTec OpticalComponents and Systems, CAPES and FAPEMIG for the support tothis research study.
References
Banks J, Carson JS, Nelson BL and Nicol DM (2005). Discrete-
Event System Simulation4th edn, Prentice-Hall: New Jersey.
Bisgaard S and Fuller H (1994). Analysis of factorial experimentswith defects or defectives as the response. Quality Eng 7(2):
429443.
Chou Y, Polansky AM and Mason RL (1998). Transforming
nonnormal data to normality in statistical process control.
J Qual Technol30: 133141.
Fu MC (2002). Optimization for simulation: Theory vs. practice.
INFORMS J Comput 14(3): 192215.
Haldar A and Mahadevan S (2000). Probability, Reliability and
Statistical Methods in Engineering Design1st edn, John Wiley &
Sons: New York.
Harrell C, Ghosh BK and Bowden R (2000). Simulation Using
ProModel3rd edn, McGraw-Hill: Boston.
Table 10 Results from the three procedures for the multi-national automotive plant application
Parameter Design of experimentsand solver
Optimizationusing four
factors
Optimizationusing allfactors
A 2 3 4
B 2 (*) 2 (*) 1C 109 (*) 109 (*) 102D 39 (*) 39 (*) 21E 105 153 126F 80 73 55G 152 (*) 152 (*) 119H 35 48 35J 74 (*) 74 (*) 54K 49 (*) 49 (*) 33Lead Time (hour) 8.59 8.28 7.42
Number of runs 64 213 2571Computational time(hour)
3 0.2 3.2 3 0.3 3.3 3.2
Note: The parameters identified with (*) were not used as input for
optimization. They were kept at low level.
14 Journal of Simulation Vol. ] ], No. ] ]
-
8/10/2019 12. Sensitivity Analysis in Discrete-event Simulation Using Fractional Factorial Designs
15/15
Kleijnen JPC (1995). Theory and methodology: Verification
and validation of simulation models. Eur J Opl Res 82:
45162.
Kleijnen JPC (1998). Experimental design for sensitivity analysis,
optimization, and validation of simulation models. In: Banks J
(ed). Handbook of Simulation 1998. John Wiley & Sons: New
York, pp 173223.
Law AM and Kelton WD (2000). Simulation Modeling and Analysis
3rd edn, McGraw-Hill: New York.Law AM and McComas MG (1998). Simulation of manufacturing
systems. In: Medeiros DJ, Watson EF and Manivannan MS
(eds). Proceedings of the 1998 Winter Simulation Conference.
Institute of Electrical and Electronics Engineers; Piscataway,
New Jersey, pp 4952.
Lewis SL, Montgomery DC and Myers RH (19992000). The
analysis of designed experiments with non-normal responses.
Quality Eng 12(2): 225243.
Montgomery DC (2001). Design and Analysis of Experiments 5th
edn, John Wiley & Sons: New York.
Montgomery DC and Runger GC (2003). Applied Statistics and
Probability for Engineers 2nd edn, John Wiley & Sons: New
York.
OKane JF, Spenceley JR and Taylor R (2000). Simulation as an
essential tool for advanced manufacturing technology problems.
J Mater Process Tech 107: 412424.
Sargent R.G (1998). Verification and validation of simulation
models. Proceedings of the 1998 Winter Simulation Conference.
Shannon RE (1998). Introduction to the art and science of
simulation. In: Medeiros DJ, Watson EF and Manivannan
MS (eds).Proceedings of the 1998 Winter Simulation Conference.
Institute of Electrical and Electronics Engineers; Piscataway,New Jersey, pp 714.
Silva WA and Montevechi JAB (2004). Verificacao do custeio de
uma ce lula de manufatura usando simulacao e otimizacao. In:
Anais do XXXVI Simposio Brasileiro de Pesquisa Operacional
SBPO: Sao Joao Del Rei, Minas Gerais. CD-ROM.
Strack J (1984). GPSS: modelagem e simulacao de sistemas. LTC:
Rio de Janeiro.
Ye KQ and Hamada M (2001). A step-down Lenth method for
analyzing unreplicated factorial designs. J Qual Technol33(2):
140153.
Received 27 September 2007;
accepted 29 July 2009 after two revisions
Montevechi et alSensitivity analysis in discrete-event simulation using fractional factorial designs 15