Prior to statistical hypothesis testing, a lot of preplanning takes place before you collect any data. This planning includes identifying the data to gather, how to collect it, and how to measure it among many other details. A crucial part of the planning is determining how much data you need to collect using which the overall duration of the test can be estimated.
Why is it important?
A pre-study calculation of the required sample size and test duration is performed in the majority of quantitative studies. This calculation helps to evaluate the cost, time, and resources needed to be deployed to detect an effect relevant to the study. The Minimum Detectable Effect (MDE) size is the smallest improvement that you want the experiment to detect in a metric (like conversion rate, the average revenue) associated with an existing asset. Performing a study without these estimates can be detrimental to the study. If the study is stopped before sufficient data, one may not be able to detect an important existing effect, whereas a prolonged study may waste time, resources, and money. Therefore, it is important to determine the optimal sample size. Moreover, calculating the sample size in the design stage of the study is increasingly becoming a necessity when seeking approval for certain research projects.
How to calculate the sample size for randomized controlled trials?
Depending on the type of study design and the study’s outcome(s), formulas for the sample size calculation can vary. These calculations are generally performed during the design of randomized controlled trials (RCTs) and depending on the primary outcome of the study, a formula is chosen.
VWO uses Bayesian metrics to compute statistical significance among the variations created on the platform. The duration calculator we use thus incorporates these metrics inside its calculation and aims to determine the number of visitors it would take to reach certain metric thresholds. You can take a free trial with VWO or request a demo from our product experts to explore this in detail.
Relationship of MDE on variability in sample size
The greater the effect size on the outcome, the smaller the sample size needed to prove it. And conversely, to determine smaller effects, it is necessary to increase the sample size.
The sample size calculation is based on assumptions about parameters of a test that can be inaccurate and is therefore subject to error. While choosing the estimates employed in calculating the sample size, one should be realistic. Highly optimistic choices about the effect size increase the risk of calculating an insufficient number of observations for the sample, whereas highly pessimistic choices can make the study unviable by resulting in a sample size that is too large to be collected in a practical scenario.
How to determine the minimum effect size?
The minimum effect size is the effect that would be meaningful for your business, so one would put in the effort to deploy the variation. In certain cases, deployment of a new variation can invite certain deployment costs, and the only way to justify those costs is when the proposed variation is satisfying a minimum uplift over the current design.
For a hypothesis test, even the tiniest effect size can be found statistically significant with a large enough sample. At first, this may seem good – just collect large enough data and find statistical significance for the tiny effect and that’s it. But is that statistical significance result useful for your business? Do you want to be making claims that an effect exists even if it’s tiny?
Good sample size calculations are based on the smallest effects that are meaningful for your business.
