A/B testing with Azure App Service deployment slots

In the world of web applications the principle of A/B testing is pretty common (https://en.wikipedia.org/wiki/A/B_testing), and for web API's this can also be used. Azure App Service has a facility called ‘development slots’ which can be used not only for zero-downtime deployments, but also for A/B testing.

What is a deployment slot and how can it help with A/B testing?

A deployment slot is a separate App Service resource hosted on the same App Service Plan. This deployment slot can be used for zero-downtime deployments which is very convenient for production deployments. When you have a deployment slot in your environment, you can deploy your code to that deployment slot. After deployment, Azure will do the warm up for you and then you can swap the two slots. This means the warmed up deployment will become the production deployment almost immediately. Having two versions of your deployment on your production environment also allows for A/B testing as you can configure the % of traffic flowing through each of the deployment slots. This is handled by the traffic manager in front of the App Service, so no additional configuration is necessary.

Let's first take a look at deployment slots in general

After you created an App Service you only have one deployment slot and having one deployment slot means that by default 100% of the traffic will flow throught it.

Out of the box deployment slot

What happens when we create one?

As mentioned earlier a deployment slot is a seperate App Service resource. On creation of the deployment slot it asks you whether you want to copy the settings of another deployment slot (details on which settings are slot specific and which are not, can be found here: https://docs.microsoft.com/en-us/azure/app-service/deploy-staging-slots#which-settings-are-swapped). At the same time the deployment slot is setup in the traffic manager of the App Service so it's ready to have traffic redirected to it.

Creating a deployment slot called ‘pre-production’ will result into the below.

Additional deployment slot

The additional App Service resource is also visible in the resource group overview.

Additional resource

So how does this A/B testing work with deployment slots?

When you open the deployment slot overview, you'll notice the “Traffic %” column.

Traffic balance

With this setting you can determine how much traffic will be directed to each of the deployment slots. By default 100% goes to the production slot (one with the green production label) and 0% to the other slots, but this can be adjused manually so you can gradually build op the consumer base using the new deployment. When there is sufficient confidence the new deployment is working fine, you can swap the deployment slots. As part of that process the traffic balance is reset and the production slot will get 100% again.

Controlled traffic routing

What if you want to have a controlled set of consumers taking advantage of new features or having a controlled set of consumers testing the new deployment? You cannot do that using the traffic % feature, as you don't know who will end up on which slot. For this you can call the App Service with a specific query parameter to instruct it to go to a particular slot: x-ms-routing-name

For example, the default URL to access the App Service is: https://test-jps-deployment-slots.azurewebsites.net

If I want to access the ‘production label’ slot I can use: https://test-jps-deployment-slots.azurewebsites.net/?x-ms-routing-name=self

and if I want to access the ‘pre-production’ slot I can use: https://test-jps-deployment-slots.azurewebsites.net/?x-ms-routing-name=pre-production

However, when you try doing that, the result is the data appears from the ‘self’ deployment slot and not from the slot named ‘pre-production.

Why does this happen?

After a close look at the documentation, it seems you need to manually key in ‘0’ in the Traffic % to kind of activate the traffic manager. When you've done that, it starts working as expected.

You need an eagle eye though to spot the difference between the default and manually keyed in value: in the second screenshot the ‘0’ is not greyed out.

Manual traffic

This option of providing the deployment slot as part of the url is a nice way to be able to test a production deployment without having it available for all consumers. Keep in mind though that this doesn't include any kind of security, so everybody who knows (or can guess) the deployment slot name is able to access it if no security measures are taken.