Page 217 - DCAP601_SIMULATION_AND_MODELING

Unit 12: Design and Evaluation of Simulation Experiments (II)

We conduct a simulation experiment, look at the results and find as many new questions as Notes
answers to our original questions. Each simulation experiment suggests subsequent simulation
experiments. Through a succession of these experiments, we gradually gain the better
understanding we originally sought. To a large extent, it is fruitful to approach simulation in the
spirit of exploratory data analysis, e.g., as in Tukey (1977), Velleman and Hoaglin (1981).
Successful simulation studies usually involve an artful mix of both experimental design and
exploration. We would emphasize the spirit of exploration, but we feel that some experimental
design can be a big help. When we plan to hike in the mountains, in addition to knowing what
peak we want to ascend, it is also good to have a rough idea how long it will take to get there:
Should the hike take two hours, two days or two weeks?
That is just the kind of rough information we need for simulations. A major purpose of simulation
experiments, often as a means to other ends, is to estimate unknown quantities of interest. When
we plan to conduct a simulation experiment, in addition to knowing what quantities we want to
estimate, it is also good to have a rough idea how long it will take to obtain a reliable estimate:
Should the experiment take two seconds, two hours or two years?
As in Whitt (1989), in this chapter we discuss techniques that can be used, before a simulation is
conducted, to estimate the computational effort required to obtain desired statistical precision
for contemplated simulation estimators. Given information about the required computational
effort, we can decide what cases to consider and how much computational effort to devote to
each. We can even decide whether to conduct the experiment at all.
The theoretical analysis we discuss should complement the experience we gain from con- ducting
many simulation experiments. Through experience, we learn about the amount of computational
error required to obtain desired statistical precision for simulation estimators in various settings.
The analysis and computational experience should reinforce each other, giving us better judgment.
The methods in this chapter are intended to help develop more reliable expectations about
statistical precision. We can use this knowledge, not only to design better simulation experiments,
but also to evaluate simulation output analysis, done by others or ourselves.
The experimental design problem may not seem very difficult. First, we might think, given the
amazing growth in computer power, that the computational effort rarely needs to be that great,
but that is not the case: Many simulation estimation goals remain out of reach, just like many
other computational goals; e.g., see Papadimitriou (1994). Second, we might think that we can
always get a rough idea about how long the runs should be by doing one pilot run to estimate the
required simulation run lengths. However, there are serious difficulties with that approach.
First, such a preliminary experiment requires that we set up the entire simulation before we
decide whether or not to conduct the experiment.
Nevertheless, if such a sampling procedure could be employed consistently with confidence,
thenthe experimental design problem would indeed not be especially difficult. In typical
simulation experiments, we want to estimate steady-state means for several different input
parameters.

Did u know? What are explorations?
The act of exploring, penetrating, or ranging over for purposes of discovery.

Unfortunately, doing a pilot run for one set of parameters may be very misleading, because the
required run length may change dramatically when the input parameters are changed. To
illustrate how misleading one pilot run can be, consider a simulation of a queueing model. Indeed,
we shall use queueing models as the context examples throughout the chapter. Now consider
the simulation of a single-server queue with unlimited waiting, with the objective of estimating
the mean steady-state (or long-run average) number of customers in the system, as a function of

LOVELY PROFESSIONAL UNIVERSITY 211