Designing Monte Carlo simulations can be a fun and rewarding experience. Whether you are interested in evaluating the performance of a new optimizer, re-evaluating previous research claims (like the ANOVA is ‘robust’ to violations of normality), determine power rates for an upcoming research proposal, or simply to appease a strange thought in your head about a new statistical idea you heard about, designing Monte Carlo simulations can be incredibly rewarding and are extremely important to those who are statistically oriented. However, organizing simulations can be a challenge, and all too often coders resort to the dreaded “for-loop” strategy, for-ever resulting in confusing, error prone, and simulation specific code. The package SimDesign is one attempt to fix these and other issues that often arise when designing Monte Carlo simulation experiments.
Generally speaking, Monte Carlo simulations can be broken into three major components:
Each operation above represents the essential components of the SimDesign package. The design component is represented by a data.frame object containing the simulation conditions to be investigated, while generate, analyse, and summarise represent user-defined functions which comprise the three steps in the simulation. Each of these components are constructed and passed to the runSimulation() function where the simulation steps are evaluated, ultimately returning a data.frame object containing the simulation results.
After loading the SimDesign package, we begin by defining the required user-constructed functions. To expedite this process, a call to SimFunctions() will create a template to be filled in, where all the necessary functional arguments have been pre-assigned, and only the body of the functions need to be modified. The documentation of each argument can be found in the respective R help files, however there organization is very simple conceptually.
To begin, the following code should be copied and saved to an external source (i.e., text) file.
library(SimDesign)
SimFunctions()#-------------------------------------------------------------------
library(SimDesign)
Design <- createDesign(factor1 = NA,
factor2 = NA)
#-------------------------------------------------------------------
Generate <- function(condition, fixed_objects = NULL) {
dat <- data.frame()
dat
}
Analyse <- function(condition, dat, fixed_objects = NULL) {
ret <- c(stat1 = NaN, stat2 = NaN)
ret
}
Summarise <- function(condition, results, fixed_objects = NULL) {
ret <- c(bias = NaN, RMSE = NaN)
ret
}
#-------------------------------------------------------------------
res <- runSimulation(design=Design, replications=1000, generate=Generate,
analyse=Analyse, summarise=Summarise)
resAlternatively, if you are lazy (read: efficient) or just don’t like copy-and-pasting, SimFunctions() can write the output to a file by providing a filename argument. The following creates a file (mysim.R) containing the simulation design/execution and required user-defined functions.
SimFunctions('mysim')For larger simulations, you may want to use two files, and if you’d prefer to have helpful comments included then these can be achieved with the singlefile and comments arguments, respectively.
SimFunctions('mysim', singlefile = FALSE, comments = TRUE)The choice of using a single file or not is entirely a matter of preference, and will not influence the overall simulation implementation.
As a toy example, let’s consider how the following question can be investigated with SimDesign:
Question: How does trimming affect recovering the mean of a distribution? Investigate this using different sample sizes with Gaussian and \(\chi^2\) distributions. Also, demonstrate the effect of using the median to recover the mean.
First, define the condition combinations that should be investigated. In this case we wish to study 4 different sample sizes, and use a symmetric and skewed distribution. The use of createDesign() is extremely helpful here to create a completely crossed-design for each combination (there are 8 in total).
Design <- createDesign(sample_size = c(30, 60, 120, 240),
distribution = c('norm', 'chi'))
Design## # A tibble: 8 × 2
## sample_size distribution
## <dbl> <chr>
## 1 30 norm
## 2 60 norm
## 3 120 norm
## 4 240 norm
## 5 30 chi
## 6 60 chi
## 7 120 chi
## 8 240 chiEach row in Design represents a unique condition to be studied in the simulation. In this case, the first condition to be studied comes from row 1, where \(N=30\) and the distribution should be normal.
We first start by defining the data generation functional component. The only argument accepted by this function is condition, which will always be a single row from the Design data.frame object of class data.frame. Conditions are run sequentially from row 1 to the last row in Design. It is also possible to pass a fixed_objects object to the function for including fixed sets of population parameters and other conditions, however for this simple simulation this input is not required.
Generate <- function(condition, fixed_objects = NULL) {
N <- condition$sample_size
dist <- condition$distribution
if(dist == 'norm'){
dat <- rnorm(N, mean = 3)
} else if(dist == 'chi'){
dat <- rchisq(N, df = 3)
}
dat
}As we can see above, Generate() will return a numeric vector of length \(N\) containing the data to be analysed each with a population mean of 3 (because a \(\chi^2\) distribution has a mean equal to its df). Next, we define the analyse component to analyse said data:
Analyse <- function(condition, dat, fixed_objects = NULL) {
M0 <- mean(dat)
M1 <- mean(dat, trim = .1)
M2 <- mean(dat, trim = .2)
med <- median(dat)
ret <- c(mean_no_trim=M0, mean_trim.1=M1, mean_trim.2=M2, median=med)
ret
}This function accepts the data previously returned from Generate() (dat), the condition vector previously mentioned.
At this point, we may conceptually think of the first two functions as being evaluated independently \(R\) times to obtain \(R\) sets of results. In other words, if we wanted the number of replications to be 100, the first two functions would be independently run (at least) 100 times, the results from Analyse() would be stored, and we would then need to summarise these 100 elements into meaningful meta statistics to describe their empirical properties. This is where computing meta-statistics such as bias, root mean-square error, detection rates, and so on are of primary importance. Unsurprisingly, then, this is the purpose of the summarise component:
Summarise <- function(condition, results, fixed_objects = NULL) {
obs_bias <- bias(results, parameter = 3)
obs_RMSE <- RMSE(results, parameter = 3)
ret <- c(bias=obs_bias, RMSE=obs_RMSE, RE=RE(obs_RMSE))
ret
}Again, condition is the same as was defined before, while results is a matrix containing all the results from Analyse(), where each row represents the result returned from each respective replication, and the number of columns is equal to the length of a single vector returned by Analyse().
That sounds much more complicated than it is — all you really need to know for this simulation is that an \(R\) x 4 matrix called results is available to build a suitable summary from. Because the results is a matrix, apply() is useful to apply a function over each respective row. The bias and RMSE are obtained for each respective statistic, and the overall result is returned as a vector.
Stopping for a moment and thinking carefully, we know that each condition will be paired with a unique vector returned from Summarise(). Therefore, you might be thinking that the result returned from the simulation will be in a rectangular form, such as in a matrix or data.frame. Well, you’d be right — good for you.
The last stage of the SimDesign work-flow is to pass the four defined elements to the runSimulation() function which, unsurprisingly given it’s name, runs the simulation. There are numerous options available in the function, and these should be investigated by reading the help(runSimulation) HTML file. Options for performing simulations in parallel, storing/resuming temporary results, debugging functions, and so on are available. Below we simply request that each condition be run 1000 times on a single processor, and finally store the results to an object called results.
res <- runSimulation(Design, replications = 1000, generate=Generate,
analyse=Analyse, summarise=Summarise)##
##
Design row: 1/8; Started: Tue Aug 30 14:44:36 2022; Total elapsed time: 0.00s
##
##
Design row: 2/8; Started: Tue Aug 30 14:44:36 2022; Total elapsed time: 0.31s
##
##
Design row: 3/8; Started: Tue Aug 30 14:44:36 2022; Total elapsed time: 0.64s
##
##
Design row: 4/8; Started: Tue Aug 30 14:44:37 2022; Total elapsed time: 0.97s
##
##
Design row: 5/8; Started: Tue Aug 30 14:44:37 2022; Total elapsed time: 1.31s
##
##
Design row: 6/8; Started: Tue Aug 30 14:44:37 2022; Total elapsed time: 1.63s
##
##
Design row: 7/8; Started: Tue Aug 30 14:44:37 2022; Total elapsed time: 1.95s
##
##
Design row: 8/8; Started: Tue Aug 30 14:44:38 2022; Total elapsed time: 2.29sres## # A tibble: 8 × 18
## sample_s…¹ distr…² bias.m…³ bias.m…⁴ bias.m…⁵ bias.m…⁶ RMSE.…⁷ RMSE.…⁸ RMSE.…⁹
## <dbl> <chr> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl> <dbl>
## 1 30 norm -4.81e-3 -4.68e-3 -2.93e-3 4.83e-3 0.185 0.190 0.196
## 2 60 norm 3.05e-3 3.92e-3 3.11e-3 5.96e-3 0.125 0.129 0.134
## 3 120 norm 1.13e-3 7.80e-4 5.72e-4 -3.26e-4 0.0908 0.0939 0.0976
## 4 240 norm 9.53e-4 1.13e-3 7.05e-4 -1.58e-4 0.0643 0.0659 0.0684
## 5 30 chi 2.22e-2 -2.99e-1 -4.44e-1 -5.77e-1 0.448 0.514 0.613
## 6 60 chi -4.28e-3 -3.46e-1 -4.94e-1 -6.36e-1 0.319 0.455 0.574
## 7 120 chi -5.02e-3 -3.46e-1 -4.88e-1 -6.35e-1 0.224 0.406 0.533
## 8 240 chi -5.83e-5 -3.46e-1 -4.89e-1 -6.32e-1 0.158 0.377 0.511
## # … with 9 more variables: RMSE.median <dbl>, RE.mean_no_trim <dbl>,
## # RE.mean_trim.1 <dbl>, RE.mean_trim.2 <dbl>, RE.median <dbl>,
## # REPLICATIONS <int>, SIM_TIME <dbl>, COMPLETED <chr>, SEED <int>, and
## # abbreviated variable names ¹sample_size, ²distribution, ³bias.mean_no_trim,
## # ⁴bias.mean_trim.1, ⁵bias.mean_trim.2, ⁶bias.median, ⁷RMSE.mean_no_trim,
## # ⁸RMSE.mean_trim.1, ⁹RMSE.mean_trim.2
## # ℹ Use `colnames()` to see all variable namesAs can be seen from the printed results, each result from the Summarise() function has been paired with their respective condition, meta-statistics have been properly named, and three additional columns have been appended to the results: REPLICATIONS, which indicates how many time the conditions were performed, SIM_TIME, indicating the time (in seconds) it took to completely finish the respective conditions, and SEED, which indicates the random seeds used by SimDesign for each condition (for reproducibility). A call to View() in the R console may also be a nice way to sift through the res object.
In this case, visually inspecting the simulation table is enough to understand what is occurring, though for other Monte Carlo simulations use of ANOVAs, marginalized tables, and graphics should be used to capture the essentially phenomenon in the results. Monte Carlo simulations are just like collecting and analysing data for experiments, so my advice would be to put on your analysis hats and present your data as though it were data collected from the real world.
In this particular simulation, it is readily apparent that using the un-adjusted mean will adequately recover the population mean with little bias. The precision also seems to increase as sample sizes increase, which is indicated by the decreasing RMSE statistics. Generally, trimming causes less efficiency in the estimates, where greater amounts of trimming results in even less efficiency, and using the median as a proxy to estimate the mean is the least effective method. This is witnessed rather clearly in the following table, which prints the relative efficiency of the estimators:
REs <- res[,grepl('RE\\.', colnames(res))]
data.frame(Design, REs)## sample_size distribution RE.mean_no_trim RE.mean_trim.1 RE.mean_trim.2
## 1 30 norm 1 1.0 1.1
## 2 60 norm 1 1.1 1.1
## 3 120 norm 1 1.1 1.2
## 4 240 norm 1 1.1 1.1
## 5 30 chi 1 1.3 1.9
## 6 60 chi 1 2.0 3.2
## 7 120 chi 1 3.3 5.7
## 8 240 chi 1 5.7 10.5
## RE.median
## 1 1.4
## 2 1.6
## 3 1.5
## 4 1.6
## 5 2.8
## 6 5.1
## 7 9.2
## 8 17.1Finally, when the \(\chi^2\) distribution was investigated only the un-adjusted mean accurately portrayed the population mean. This isn’t surprising, because the trimmed mean is, after all, making inferences about the population trimmed mean, and the median is making inferences about, well, the median. Only when the distributions under investigation are symmetric are the statistics able to draw inferences about the same inferences about the mean of the population.
The following is a conceptual breakdown of what runSimulation() is actually doing behind the scenes. Here we demonstrate the results from the first condition (row 1 of Design) to show what each function returns.
A single replication in a Monte Carlo simulation results in the following objects:
(condition <- Design[1, ])## # A tibble: 1 × 2
## sample_size distribution
## <dbl> <chr>
## 1 30 normdat <- Generate(condition)
dat## [1] 2.37 3.18 2.16 4.60 3.33 2.18 3.49 3.74 3.58 2.69 4.51 3.39 2.38 0.79 4.12
## [16] 2.96 2.98 3.94 3.82 3.59 3.92 3.78 3.07 1.01 3.62 2.94 2.84 1.53 2.52 3.42res <- Analyse(condition, dat)
res## mean_no_trim mean_trim.1 mean_trim.2 median
## 3.1 3.2 3.2 3.3We can see that Generate() returns a numeric vector which is accepted by Analyse(). The Analyse() function then completes the analysis portion using the generated data, and returns a named vector with the observed parameter estimates. Of course, this is only a single replication, and therefore is not really meaningful in the grand scheme of things; so, it must be repeated a number of times.
# repeat 1000x
results <- matrix(0, 1000, 4)
colnames(results) <- names(res)
for(i in 1:1000){
dat <- Generate(condition)
res <- Analyse(condition, dat)
results[i, ] <- res
}
head(results)## mean_no_trim mean_trim.1 mean_trim.2 median
## [1,] 3.1 3.1 3.1 2.9
## [2,] 3.1 3.1 3.1 3.1
## [3,] 3.1 3.1 3.0 2.8
## [4,] 2.7 2.6 2.6 2.7
## [5,] 3.2 3.2 3.2 3.0
## [6,] 3.1 3.1 3.0 3.1The matrix stored in results contains 1000 parameter estimates returned from each statistic. After this is obtained, we can move on to summarising the output through the Summarise() function to obtain average estimates, their associated sampling error, their efficiency, and so on.
Summarise(condition, results) ## bias.mean_no_trim bias.mean_trim.1 bias.mean_trim.2 bias.median
## -0.0011 -0.0031 -0.0035 -0.0037
## RMSE.mean_no_trim RMSE.mean_trim.1 RMSE.mean_trim.2 RMSE.median
## 0.1739 0.1777 0.1859 0.2146
## RE.mean_no_trim RE.mean_trim.1 RE.mean_trim.2 RE.median
## 1.0000 1.0442 1.1425 1.5225This scheme is then repeated for each row in the Design object until the entire simulation study is complete.
Of course, runSimulation() does much more than this conceptual outline, which is why it exists. Namely, errors and warnings are controlled and tracked, data is re-drawn when needed, parallel processing is supported, debugging is easier with the debug input (or by inserting browser() directly), temporary and full results can be saved to external files, the simulation state can be saved/restored, build-in safety features are included, and more. The point, however, is that you as the user should not be bogged down with the nitty-gritty details of setting up the simulation work-flow/features; instead, you need only focus your time on the important generate-analyse-summarise steps, organized in the body of these function, required to obtain your interesting simulation results.
To access further examples and instructions feel free to visit the package wiki on Github