How to predict the required duration of an A/B split test

[nusaiba127]

5.
Evanmiller.org has another handy conversion optimization tool called the sample size calculator .

This tool allows you to answer the question "How long will it take to get reliable test results?" and it is not an answer you should try to guess.


298-8.jpg

There are a few things to note. First, the tool has south africa consumer email list an absolute/relative toggle — if you want to find out the difference between a base conversion rate of 5% and a variable conversion rate of 6%, it will be 1% in absolute terms (6-5=1) or 20% in relative terms (6/5=1.2). Second, at the bottom of the page there are two “sliders”. The bottom one is responsible for the desired significance level — if your goal is to achieve 95% significance, then the slider should be set to 5%. The top slider shows the probability that the number of required page visits will be sufficient — for example, if you want to know the number of visits needed to achieve an eighty percent chance of finding 95% significance, set the top slider to 80% and the bottom one to 5%.

6. What not to do
There are a few simple ways to detect the failure of a split test, although they are not always obvious at first glance:

A) Split testing of non-binary ordinal values

For example, your goal is to find out whether there is a significant difference in the probability that visitors from the “original” and “after changes” groups will buy certain products. You label three products “1”, “2”, and “3”, and then enter these values ​​into the significance test fields. Unfortunately, this approach won’t work – product 2 is not the average of products 1 and 3.