Skip to the content

How to Run an A/B Test and Evaluate the Results

How to A/B Test Like a Boss

“A/B testing is widely used, because [the tests] are easy to deploy, easy to understand, and easy to explain to management” – Christopher Manning, professor of Computer Science

Since A/B testing is widely used, reliable, and often simple to explain, there’s a belief that the tests will run themselves and give you quick, actionable results. This sentiment is a little misguided. The reality is that your testing is only as effective as your planning and evaluating skills. It’s like trying to read a classic novel in a different language: if you don’t understand how to read what you’re looking at, then it’s useless.

Despite being relatively simple in theory, in practice AB tests become deeply complex, especially if you want to get the most out of them. On top of that, to get the most out of testing in Umbraco, you’ll probably need some outside help from programs that further complicate the issue (we’ll cover that later, too). For now, let’s discuss how to run an A/B test and evaluate the results to understand the complexities better and to maximize your returns and convert the most customers, or, to put it more eloquently, let’s learn how to A/B test like a boss.


A/B testing and the scientific method

A/B testing is a lot like the scientific method in that it stems from a hypothesis formed to solve a problem, and from there it collects empirical evidence from observing an experiment. A well-planned A/B test is not just a random battle of two pages with opposite designs, it’s a targeted and very specific method for testing the effect of changes to a chosen variable and examining how it influences the behaviour of visitors.

Having a concrete hypothesis in place before starting is like building a strong foundation for a home: you can make random changes and throw a test together just the same as you can build a few walls and a roof, but without being framed properly and rooted in solid ground, both won’t be of much use for very long. For example, to frame your test you could start with “I think the Checkout button needs to be made more visible, because a high % of visitors reach the Checkout page but most close out instead of completing the purchase”. In order to begin a successful test, you need:


  • A goal - I want to increase conversions by 10%.
  • A problem statement - a small % of visitors that reach the Checkout page are actually buying.
  • A hypothesis - since visitors are reaching the page but not clicking the Checkout button, it could be because the button is not visible enough. Increasing its visibility will increase the number of clicks.


Planting solid roots will give you firm ground to stand on when it comes to evaluating the potential impact of changes to your site.


Think broadly and keep an open mind

“The problem is rooted in the fact that most people overvalue how good their website already is” -Dan Siroker, CEO of Optimizely

Before beginning testing, it’s important to keep an open mind. Companies should resist the temptation of pursuing only incremental changes before broadly exploring possible solutions. The issue is that if you focus on solely refining one aspect of your site, you might miss the solution that actually provides the most dramatic results. More simply, take the time to explore multiple hypothesis before focusing in on one and refining it.

A good real-world example of this is when the ABC Family Channel started A/B testing in the hopes of increasing engagement rates. They originally went in hoping to increase engagement for visitors looking for new shows, only to realize after exploration that people weren’t coming to the site for that purpose at all. Further exploration showed that visitors were coming to the site to watch episodes of shows that they had missed. They arrived at a design that was optimized for the majority of their traffic and increased engagement by 600% (reference)!

Next up: How to Choose What to Test