How Many Simulations Should I Run to Know That I've Run Enough?

Say that five times fast. Seriously, though, how many simulations are enough to be confident that your results are accurate?

Edited 2-11-2015 for clarity

DCS recommends running 5000 to 20,000 simulations when analyzing a model.

Here is why:

Statistics are estimates of the parameters of a population. 3DCS results are statistics based on a sample (the number of simulations run) of an infinite population (the number of simulations that could be run). Because a statistic is an estimate, the confidence interval is used to determine how good an estimate it is. The confidence interval is calculated from the sample's size and standard deviation and the chosen confidence level (typically 90%, 95%, or 99%).

After a simulation has been run, a user may look at the confidence interval to determine if enough samples have been run.

For example, a measure may have a standard deviation of 1 mm after 5000 simulations were run. The 95% confidence interval will be from 0.981 to 1.020 mm. Therefore, assuming the measure results are normally distributed, a user has a 95% probability of generating a standard deviation between 0.981 and 1.020 mm if they let the model run forever.

Because the confidence interval is linearly related to the standard deviation, a standard deviation of 2 mm would have a 95% confidence interval from 1.962 to 2.040 mm and any simulation of 5000 samples and a standard deviation of x mm would have a 95% confidence interval from 0.981*x to 1.020*x mm. The user would be confident the results are with 2% of the population standard deviation.

To be more confident, more samples are needed. To be confident the results are with 1% of the population standard deviation, 20,000 simulations are needed.

Running even more samples will narrow the confidence interval further, but since many other factors affect model accuracy, running more than 20,000 samples generally will not give the user more accurate results.

What is a Confidence Interval?

Click here to take a look at our previous post on Confidence Intervals

Want to see where these numbers came from? Click here for the third and final part in our Confidence Interval series.

Visit the DCS Community to learn more