Skip to the content

Running A/B Experiments in Umbraco Like a Pro with uSplit

We have recently released a new version of uSplit that brings unparalleled A/B testing experience to the Umbraco CMS. It combines Umbraco’s familiar and intuitive user interface with the powerful analysis and reporting capabilities of Google Analytics. This article demonstrates the features of uSplit and shows how easy A/B testing in Umbraco has become.

Ensuring uSplit is ready to use

Before we start, we need to make sure uSplit is installed and configured correctly. Don’t have uSplit yet? Check out the product page and the installation guide.

uSplit integrates with Google Analytics – it creates a Google Analytics Experiment in the background and reports rendered variations to your Google Analytics account. In order to do so, uSplit needs to obtain permission from someone with access to the desired Google Analytics account. This authorization flow is invoked from the A/B testing dashboard (in the Content section). This only needs to happen once for each site – a special token is persisted and further communication with Google Analytics is done on behalf of the person granting the permissions. A screenshot below shows a dashboard of a properly configured plugin.

Experiments overview

Currently uSplit allows you to create A/B experiments for individual pages. When such a page is accessed, for each active experiment uSplit picks a variation for the visitor (based on the recommended algorithm), applies any modifidations, and renders the modified content to the visitor. There are no redirects involved – the page under test is modified on the server side and served under the same URL. The user does not know they are subject to an experiment.

It is not possible to span a single experiment across multiple pages. This can be added in the future if there is demand.

For each experiment you will define several variations. A variation is simply a new page of the same Document Type as the page under test. When a variation is selected to be shown to the visitor, uSplit overrides the properties of the original document with values from this variation; when you leave a property empty the original will be used. It is also possible to change the template by setting a different template for the variation.

Since the variations are just new instances of the same page, you get the same experience you are used to when editing them. Furthermore, it does not matter where the variations are stored, so you can hide them in a separate folder that is not rendered or publicly accessible – it does not affect the way your menus work.

Finally, by default uSplit reports the selected variation to Google Analytics by embedding a small piece of JavaScript right after the head tag (expects you to use analytics.js). This can however be tweaked to match your Google Analytics scenario, see the installation guide for details.

Creating and running your first experiment

Start by right-clicking on the A/B testing node on the Content section and selecting Create a new Experiment. Then, select which content item you want to A/B test.

You will be presented with an experiment details screen. The experiment itself has also been registered with Google Analytics and you can manage it there as well, tweaking advanced properties (such as the metric) not available from within Umbraco.

Before you can run your experiment, you need to define and set up the variations.

Adding variations

Variations are created either by Cloning the original page, or by adding an existing page as a variation. As mentioned earlier, each page linked as a variation just stores data that is then substituted for the original content when that variation is chosen. The variation pages can therefore be located anywhere in your content tree. It is advisable to create a new folder in the root of your site and store your variations there. See section bellow on how to create such a folder.

When you click on the Clone original button, a dialog will appear, prompting you for the location under which you want the copy to be saved. You can move the variations after you create them. uSplit saves the IDs of the nodes, it does not depend on the actual location of the nodes in the tree.

Keep in mind that if you save the copy next to the original page, your menus might show both pages. That is why we advise to save the variations to a separate branch in the content tree.
Normal restrictions about content placement still apply, when you are cloning a page under a document make sure that Document Type of the original page is allowed under the target node.

If you decide to create the variation pages manually, you can later link to them by clicking on the Add existing page button. This will show a content dialog prompting you to select the existing page you want to use as a variation. Make sure you only select pages of the same Doc Type.

It is important that all the selected variations are published.

Creating a variations folder (optional)

We suggest creating a dedicated “folder” under your site root to store your variations at a central place. Doing this will allow you to move the variations out of harms way – being accidentally rendered in your menus or confusing your editors. The most important benefit is being able to block robots from accessing variations by blocking the parent folder, and allowing you to create 404 redirects when accessed directly.

To achieve the above simply create a special Document Type with no properties – simply to act as a container for your variations. Create an instance of this document type in the root of your site (just a suggestion). You may now move all your variations there and create any new ones under this folder.

Standard content placement rules still apply. Make sure you allow all document types as children of the variations folder, and allow it to be created at root. Check the screenshot below for an example.

Editing variations – what can be varied

Since variations are just ordinary pages of the same document type as the original page, you can vary any property that you can normally edit when you edit an instance of such document type.

In addition to ordinary properties we also support changing the template – simply assign a different template to the variation and if it is chosen that template will be used instead.

What is not changed

The context of the original node is preserved. If you run any content queries in your template, such as asking for the Children of the current page, or for its parents, these will execute in the context of the parent node. We also do not override the medatada, such as the ID, author or name.

In the future we would like to give you the option to override the child items as well. You could either inherit the original children (which is the current behaviour), or, if you defined any children under the variation, those would be used instead. Such a feature would be useful in scenarios where you use the children collection to source data for list of items under the current page – say show a list as offers.

Running an experiment

When you are ready to run your experiment simply click on the Commence experiment button. If you have not set any custom metric in Google Analytics, we will use Pageviews as the default metric. The metric is used to evaluate the performance of each variation which affects the probability a particular variation is chosen over the course of the experiment.

Initially each variation (including the original page, which is a valid variation in itself) has the same probability. As more data comes in the metrics are used to adjust the probabilities and some variations might be deactivated completely. This is done automatically twice a day by Google Analytics Experiments and uSplit refreshes the experiment data periodically to use the updated probabilities when it selects the variations.

Evaluating the results

Eventually one variation might be chosen as the winner and the experiment will end. This only happens when one variation is performing above the threshold which you can manually tweak in Google Analytics for each experiment as well. You can also end the experiment manually without waiting for the winner to be chosen.

You can get much more details about the experiment run from inside Google Analytics. We recommend you use Google Analytics exclusively to analyse the data.