Statistical significance means that your experiment results are likely not due to random chance.
Any experiment seeks to prove that the results from the sample also exist in the overall population. The more visitors you have, the easier it is to disprove randomness.
Before starting an experiment, you generally have a hypothesis about what will make your website better. It might be changing text, adjusting layouts, or updating colors. SplitWit evaluates your hypothesis based on the metrics you set for visitors. Our algorithm ensures that the experiment is not considered complete until the results are statistically significant.
How does SplitWit use statistical significance?
You can check the results of a SplitWit experiment at any time. The results page tells you which version is performing better based on the metrics you have set. It also shows wether these results are statistically significant, or not.
Your experiment is not considered finished until statistical significance has been achieved. But, what does that mean?
It means there is a 95% chance that these results are real, and not due to a false positive. SplitWit uses formulas to calculate this number based on probability. This calculation gives you confidence that the changes you make to your website will positively impact your business.