Running A/B Experiments in Umbraco Like a Pro with uSplit
We have recently released a new version of uSplit that brings unparalleled A/B testing experience to the Umbraco CMS. It combines Umbraco’s familiar and intuitive user interface with the powerful analysis and reporting capabilities of Google Analytics. This article demonstrates the features of uSplit and shows how easy A/B testing in Umbraco has become.
Ensuring uSplit is ready to use
uSplit integrates with Google Analytics – it creates a Google Analytics Experiment in the background and reports rendered variations to your Google Analytics account. In order to do so, uSplit needs to obtain permission from someone with access to the desired Google Analytics account. This authorization flow is invoked from the A/B testing dashboard (in the Content section). This only needs to happen once for each site – a special token is persisted and further communication with Google Analytics is done on behalf of the person granting the permissions. A screenshot below shows a dashboard of a properly configured plugin.
Currently uSplit allows you to create A/B experiments for individual pages. When such a page is accessed, for each active experiment uSplit picks a variation for the visitor (based on the recommended algorithm), applies any modifidations, and renders the modified content to the visitor. There are no redirects involved – the page under test is modified on the server side and served under the same URL. The user does not know they are subject to an experiment.
For each experiment you will define several variations. A variation is simply a new page of the same Document Type as the page under test. When a variation is selected to be shown to the visitor, uSplit overrides the properties of the original document with values from this variation; when you leave a property empty the original will be used. It is also possible to change the template by setting a different template for the variation.
Since the variations are just new instances of the same page, you get the same experience you are used to when editing them. Furthermore, it does not matter where the variations are stored, so you can hide them in a separate folder that is not rendered or publicly accessible – it does not affect the way your menus work.
Creating and running your first experiment
Start by right-clicking on the A/B testing node on the Content section and selecting Create a new Experiment. Then, select which content item you want to A/B test.
You will be presented with an experiment details screen. The experiment itself has also been registered with Google Analytics and you can manage it there as well, tweaking advanced properties (such as the metric) not available from within Umbraco.
Before you can run your experiment, you need to define and set up the variations.
Variations are created either by Cloning the original page, or by adding an existing page as a variation. As mentioned earlier, each page linked as a variation just stores data that is then substituted for the original content when that variation is chosen. The variation pages can therefore be located anywhere in your content tree. It is advisable to create a new folder in the root of your site and store your variations there. See section bellow on how to create such a folder.
When you click on the Clone original button, a dialog will appear, prompting you for the location under which you want the copy to be saved. You can move the variations after you create them. uSplit saves the IDs of the nodes, it does not depend on the actual location of the nodes in the tree.
Normal restrictions about content placement still apply, when you are cloning a page under a document make sure that Document Type of the original page is allowed under the target node.
If you decide to create the variation pages manually, you can later link to them by clicking on the Add existing page button. This will show a content dialog prompting you to select the existing page you want to use as a variation. Make sure you only select pages of the same Doc Type.
It is important that all the selected variations are published.
Creating a variations folder (optional)
We suggest creating a dedicated “folder” under your site root to store your variations at a central place. Doing this will allow you to move the variations out of harms way – being accidentally rendered in your menus or confusing your editors. The most important benefit is being able to block robots from accessing variations by blocking the parent folder, and allowing you to create 404 redirects when accessed directly.
To achieve the above simply create a special Document Type with no properties – simply to act as a container for your variations. Create an instance of this document type in the root of your site (just a suggestion). You may now move all your variations there and create any new ones under this folder.
Editing variations – what can be varied
Since variations are just ordinary pages of the same document type as the original page, you can vary any property that you can normally edit when you edit an instance of such document type.
In addition to ordinary properties we also support changing the template – simply assign a different template to the variation and if it is chosen that template will be used instead.
What is not changed
The context of the original node is preserved. If you run any content queries in your template, such as asking for the Children of the current page, or for its parents, these will execute in the context of the parent node. We also do not override the medatada, such as the ID, author or name.
Running an experiment
When you are ready to run your experiment simply click on the Commence experiment button. If you have not set any custom metric in Google Analytics, we will use Pageviews as the default metric. The metric is used to evaluate the performance of each variation which affects the probability a particular variation is chosen over the course of the experiment.
Initially each variation (including the original page, which is a valid variation in itself) has the same probability. As more data comes in the metrics are used to adjust the probabilities and some variations might be deactivated completely. This is done automatically twice a day by Google Analytics Experiments and uSplit refreshes the experiment data periodically to use the updated probabilities when it selects the variations.
Evaluating the results
Eventually one variation might be chosen as the winner and the experiment will end. This only happens when one variation is performing above the threshold which you can manually tweak in Google Analytics for each experiment as well. You can also end the experiment manually without waiting for the winner to be chosen.
You can get much more details about the experiment run from inside Google Analytics. We recommend you use Google Analytics exclusively to analyse the data.