@Segio: this means that if you don’t see at least 50% of performance difference within X amount of days, you should give up on the test. The improvement can be seen any time during day 0 – day X but if it isn’t seen even after day X, you should decide that the variation is performing equivalently to control (within confidence level you have set).

@Quan: I’m afraid I don’t know the numerator for 80% alpha. Would have to do a literature search for that.

What if I wanted to drop the Confidence Level 80% (alpha) with 80% Power (beta)? Do you know what the Numerator for Sample Size Equation would be and what would be the pitfall for doing that?

Reason I ask is my boss wanted to test to a lower confidence, but I could only adjust the Power using table 2.1 in the Statistical Rules of Thumb link you provided.

@Lori: I think this paper from Microsoft has a formula for 90% power http://www.exp-platform.com/Pages/hippo_long.aspx

Thank you for providing this great A/B test duration calculator!

I am also interested in getting the duration estimates when the chance of finding a difference is 90% or 99%.

I found the 90% duration estimate by using Table 2.1 in the book you reference on this page, i.e in the chapter from “Statistical Rules of Thumb” at http://www.vanbelle.org/chapters/webchapter2.pdf (the two sample numerator is 21). And I used the formula in your spreadsheet to add a few columns to display the 90% values.

Could you please tell me what the missing row with Power = 0.99 from Table 2.1 is, or point me to a reference that has this information.

Better yet, it would be great to add these extra 2 “confidence levels” to your calculator!

Thanks,

Lori Proos