Numbers (Photo credit: RichardBowen)

What is A/B testing?

A/B testing shows two different versions of a concept, page, or element to different users on a randomized basis to determine how each version compares on a variety of metrics.

What is multivariate testing?

Multivariate testing expands on A/B to include numerous versions, and often includes several individual tests at once.

Why would I use A/B and multivariate testing?

A/B and multivariate testing are heavily used by marketing teams in the web and application development industry. These techniques are very useful to provide a final, quantitative answer for stakeholders. Marketers primarily focus on conversion rate; user experience researchers can look at a variety of metrics along with conversion to determine advantages and disadvantages for each version.

What are the limits to A/B and multivariate testing?

A/B and multivariate testing are quantitative techniques. For the user experience researcher, these techniques are most often used near the end of product or feature launch, and for ongoing optimization.

A/B and multivariate testing, as a result of being quantitative, will never tell you precisely why a specific version had the effect it did. Because of this fact, user researchers tend to use A/B and multivariate testing after conducting other research methods. A/B and multivariate testing are also often combined with other methods; such as surveys and usability tests. These combination studies can often reduce the expense of A/B and multivariate testing.

As a purely quantitative method by itself, A/B and multivariate testing often takes a very large quantity of users to reach statistical significance. Plan on a minimum of 10,000 users per version.

A/B and multivariate testing carries more risks than other methods because it is usually conducted with live, non-opt-in users in order to gain sufficient numbers. As a result, this technique, while free to inexpensive to set up, can be more cost than most.

Ensure, at a minimum, that any results you use are at least 98% statistically significant. Noise is a very real problem in A/B and multivariate testing. Several calculators for statistical significance are available online.

How do I conduct A/B and multivariate testing?

Google Analytics, along with custom event reporting, is the most commonly used tool. A/B and multivariate testing is also easy to set-up using just a database and a few lines of computer code; just make sure you record all relevant events if you go ad-hoc. Other premium tools are also available, such as Optimizely and Visual Website Optimizer.