Reporting - Bayesian test

Bayesian tests enable you to calculate the confidence interval of a gain, as well as the median value. They enable you to find out the extent of the potential risk related to putting a variation into production following a test and are accessed via the Confidence interval tab of the new reporting layout.

We recommend following three business rules before making a decision after running a test: - wait until you have recorded at least 5,000 unique visitors per variation; - let the test run for at least 14 days (two business cycles); - wait until you have reached 300 conversions on the main goal.

Principle

The Bayesian test (or Bayesian statistic) stems from the calculation method developed by mathematician Thomas Bayes. It is based on known events, such as the variation conversion rate. The advantage of Bayesian tests is that they provide the confidence interval of the gain as well as its median value.

The Bayesian approach provides further details on gain probability, which only measures a confidence indicator, i.e. the likelihood of a strictly positive gain. We estimate that if the gain likelihood is equal to or higher than 95%, the variation can be implemented. However, this method isn't precise and does not provide information on the nature of the risk. Indeed, even with a gain probability equal to or higher than 95%, it is important to analyze results closely to avoid misinterpretation. Bayesian tests enable sound and fast decision-making and guarantee the reproducibility of results (statistical reliability).

The confidence interval enables you to identify the lowest and the highest gain markers you can achieve in 95% of cases. This indicator enables you to manage uncertainty related to conversion rate measurements. The smaller the intervals, the lower the level of uncertainty.
The confidence interval of the gain provides a bracket of what you can actually hope to achieve by replacing the original version with the variation. That is why this measurement is critical for decision-making.

The improvement rate indicates an average value, which is why there is a slight difference compared to the median value calculated by the Bayesian test.

Decision-making

Decision-making depends on the results shown on the reporting page.

If the results are clear-cut and quickly reveal overperformance (i.e. gain probability over 95% and a positive confidence interval), you can stop your test and run the changes in production to quickly benefit from these gains; as long as the 14-day rule is complied with.

If the results are less clear-cut, i.e. the interval is wide, you can decide to wait until you have gathered more information before fine-tuning your decision while taking the costs of implementing the changes into account.

Finally, if the gain probability is strictly positive but its measurement remains imprecise (with a wide confidence interval between +5% and +50%, for instance), while the cost of implementing the changes is low, you can implement the variation without taking a significant risk.
If, on the other hand, the implementation cost is high, you should wait until more information has been collected for a more precise measurement. When confidence intervals are small, you can assess the relevance of putting into production as a function of implementation costs.

Calculation method

The calculation method for Bayesian tests varies depending on the selected view. However, Bayesian tests are only applicable for “conversion” type goals, i.e. if the answer to the question is “yes” or “no”. For “value” type goals (such as the average cart), the Bayesian test cannot be calculated.

 

Visitor view

In the Visitor view, the calculation is based on the confidence intervals of the Bayesian test.
The indicator is found in the Advanced tab of a goal, which is displayed by default.
We recommend viewing the advanced statistics in addition to gain probability.

 

Sessions view

The Bayesian test cannot be calculated in the Sessions view. The decision may therefore only be based on gain probability.

Interpretation

Reading results

Confidence intervals can take values from negative to positive infinity and are rounded to the nearest hundred.

• The more the intervals fall within positive values, the higher the gain will be.
• The more the intervals fall within negative values, the higher the loss will be.

To determine the worst-case scenario, the main indicator is the lower marker of the interval. If this marker is positive, a gain will be recorded in 97,5% of cases. On the other hand, if the upper marker of the interval is negative, a loss will be recorded in 97,5% of cases.

 

Examples

Case #1: High gain

In this example, the chosen goal is the CTA click rate (action tracking) in the Visitor view. The A/B test includes three variations.
The conversion rate of variation 2 is 34,1%, compared to 25% for the original version. The increase in conversion rate compared to the original version equals 36,4%.
The confidence interval of the Bayesian test displays a marker below +2,25% for variation 2 and a marker above +48,4%, as well as a median at +36,4%. 
The median means there is a 50% chance of the gain percentage lying above 36,4% and a 50% chance of it lying below 36,4%.
In 95% of cases, the gain will lie between +2,25% and +48,40%. 

There remains a 2,5% chance of the gain lying below 2,25% and a 2,5% chance of it lying above 48,40%.

Related articles

New reporting - Test campaigns