An A/B test, also known as a split test, is an experimental process where marketers divide their target audience into two groups (A and B) and show them different versions of a certain image, webpage, campaign, email or other content. One group is exposed to the original version and the other group is exposed to the new version. The goal of this process is to test how one version performs alongside another.
Let’s say you want to improve the click-through rate (CTR) of an important button on your homepage and (after a bit of analysis) think the colour of this button could play a crucial role in this. To run an A/B test, you’ll need to create two different versions of your homepage: a “control” homepage where the button has its original colour, green for example, and a new “test” homepage where you change the colour of the button to red. You’ll then split the visitors of your homepage into two similarly sized audiences and show them either one of the versions. After a predefined period of time you should be able to analyze whether there is a statistically significant difference between the two button colours or not. Suppose the CTR of the new red button is significantly higher than the red one, then the marketers should strongly consider changing the button colour to red.
Note that in this example we only changed one single variable: colour. If we would also change the position and size of the button on our test homepage, we wouldn’t be able to attribute the outcome of our test solely to the colour. In order to test changes in multiple variables, we could conduct a multivariate test. Tip: if you don't know what element has most effect on a website, you can use a multivariate test to find out, and then use A/B testing to optimize that element.
It’s also possible to involve more than two variations for one variable in your test. Assume you would also create versions of your homepage with an orange or yellow button. Or maybe you even want to test 41 different shades of blue. As long as you only change one single variable in your versions, the A/B test principles would still apply. We call this an A/B/n test, where “n” refers to the extra number of variations being tested.