subject: Calculating Statistical Significance for Your A/B Tests [print this page] Calculating Statistical Significance for Your A/B Tests
An A/B test allows you to choose the best headline, layout, order button or another element for your web page. You set up a test where you display two variants (A and B) to different website visitors in turns. You wait for a while. It turns out, after 200 visitors, that variant A produced 10 orders out of 100 visitors and the variant B produced 12 out of 100.
So what's next? Do you declare B the winner, adopt the changes and move on? If you learned a little about testing and statistics, you know that before drawing any conclusions, you need to determine the statistical significance of the result.
Spoiler: in this case, there is a 75% probability that similar results could be obtained from two identical pages by pure chance alone. For variant B to beat variant A with 95% probability, it should have produced at least 20 sales. 16 sales would mean a 75% chance that the result is not random. Anything less, you can almost as well flip a coin to choose the winner. Think about these figures and bear them in your mind during your future tests, even if you won't calculate anything.
One more thing to keep in mind is that for the statistical analysis to even work at all, you need at least 10 results (sales, clicks, etc.) for each variant. Smaller numbers carry too much random noise to analyze them.
So where do these statistical significance figures come from, and how do you calculate them for your own tests? I will not write the exact formulas here (they are fairly complicated), but I'll give you some hints and directions so you can find them if you really want to.
The principle behind these calculations is to find the "chi-square" value for your experimental results, and to compare it to the known chi-square values for the random distribution. If your value is higher than 3.84, for example, that means there is a 95% chance that your results were not purely random. This is called a Pearson's Chi-square test. The generic formula may scare you if you don't deal much with mathematics, but for a simple A/B test you can use a simplified form, with one degree of freedom and a 22 contingency table. Look it up in a statistics course book, or even on Wikipedia.
If all of that sounds too complicated and tiresome, I have good news for you. A lot of people faced this problem, and there are tools available to make all calculations for you. Some of them are downloadable software that you need to install on your computer, and some are online tools accessible through your web browser.
