Protecting Azure Service bus topics and subscriptions via API Management

Azure service bus is one of the services Microsoft has identified as part of the Integration Services and is an important component in messaging solutions. It can be interacted with using a variety of methods, like via the SDK or a REST endpoint. One of the other key integration services is Azure API management (APIM), and its used for centralizing endpoint management. In an integration landscape, also service bus endpoints should be exposed via API management so we can provide a consistent way of accessing endpoints to clients. One of the other main advantages of APIM is, we're able to relocate (or even replace) service bus topics, without impacting clients as they only know about the APIM endpoint. So clients who want to post for example an order message, just use the message bus orders endpoint, not interested in where it eventually ends up.

From a client point of view, it's important to have a predictive experience when consuming endpoints, so they don't have to adopt a new way of accessing an endpoint every time. Also we like to abstract endpoint integration details in API management, so clients don't need to know things like service bus access keys or mandatory HTTP headers. Having security details abstracted, allows for example for key rotation without impact on the clients. We also can provide a consistent experience to the clients regarding endpoint security. When all endpoints are secured with OAuth2 or AAD JWT, it's clear for clients what authentication mechanism is required.

In this blog post we use Terraform as our way to generate and configure the resources in Azure. You can follow along by checking out this repo.

For those not familiar with Terraform, here a small crash course:

  • Start with downloading the latest Terraform version, extract and copy terraform.exe into the servicebus folder.
  • Open the repo in Visual Studio Code and open a terminal window in the servicebus folder.
  • Use az login to make sure you're in the correct Azure subscription.
  • Next we need to initialize Terraform, do this by executing terraform init.
  • Finally, run terraform apply -var-file=servicebus.dev.tfvars, this will show you what Terraform is going to do and what will be the end-result, key in ‘yes’ to start making the necessary changes

Setup Service Bus

The first thing we need is a Service bus namespace, and this has to be a Standard tier instance because we need topics and Standard tier is the lowest tier supporting that. With that in place, we need to setup permissions. To be able to read-and-delete from topic subscriptions, we need manage/send/listen permissions. We have the option to specify security policies on service bus and topic level, and in this example it's set on service bus level.

We also need a topic to receive the message on, for this example we create an orders and customers topic, and on the topics we create a subscription.

This results in the following Terraform script (variables are defined in separate tfvars file, see “repo link”).

# Create a new resource group
resource "azurerm_resource_group" "rg" {
  name     = local.resourceGroupName
  location = var.location
  tags     = var.tags
}

# Create service bus namespace
resource "azurerm_servicebus_namespace" "sb" {
  name                = "didago-processing"
  location            = azurerm_resource_group.rg.location
  resource_group_name = azurerm_resource_group.rg.name
  sku                 = "Standard"
}

# Create SAS policy
resource "azurerm_servicebus_namespace_authorization_rule" "authRule" {
  name                = "messagebus-policy"
  resource_group_name = azurerm_resource_group.rg.name
  namespace_name      = azurerm_servicebus_namespace.sb.name
  listen              = true
  send                = true
  manage              = true
}

# Create orders topic on service bus
resource "azurerm_servicebus_topic" "sbOrdersTopic" {
  name                = "orders"
  resource_group_name = azurerm_resource_group.rg.name
  namespace_name      = azurerm_servicebus_namespace.sb.name

  enable_partitioning = true
}

# Create customers topic on service bus
resource "azurerm_servicebus_topic" "sbCustomersTopic" {
  name                = "customers"
  resource_group_name = azurerm_resource_group.rg.name
  namespace_name      = azurerm_servicebus_namespace.sb.name

  enable_partitioning = true
}

# Create orders subscription on topic
resource "azurerm_servicebus_subscription" "sbOrdersSubscription" {
  name                = "orders"
  resource_group_name = azurerm_resource_group.rg.name
  namespace_name      = azurerm_servicebus_namespace.sb.name
  topic_name          = azurerm_servicebus_topic.sbOrdersTopic.name
  max_delivery_count  = 1
}

# Create customers subscription on topic
resource "azurerm_servicebus_subscription" "sbCustomersSubscription" {
  name                = "customers"
  resource_group_name = azurerm_resource_group.rg.name
  namespace_name      = azurerm_servicebus_namespace.sb.name
  topic_name          = azurerm_servicebus_topic.sbCustomersTopic.name
  max_delivery_count  = 1
}

We now have the basic setup:

  • service bus namespace
  • defined a security policy
  • two topics
  • subscription per topic

Generate Service bus SAS token

With the service bus configuration in place, it's time to look at the security side of things. As mentioned before, we like to abstract the security details for the clients, so they don't need to know the key to access service bus. If we want to send or receive messages via the service bus endpoint, we need to provide an SAS token. This SAS token is generated based on the url, the access policy key and policy name.

For most resources we can get configuration details via Terraform data resources. The SAS token cannot be retrieved from one of the resources in Terraform, so we need to generate it as part of the provisioning. How to generate this token is documented on Microsoft docs. This now needs to be embedded in the Terraform script. We want to generate this as part of the provisioning, to use it when importing the endpoint in API management.

Running Powershell from Terraform can be challenging, but fortunately Michael Stephenson wrote an excellent blog post covering this exact part. I gladly borrowed some parts of his solution.

To start with, we need to use the external data source, which allows us to execute a script, Powershell in our case. We can feed it wiht parameters by using the query parameter.

# Generate SAS token to be used in the APIM policy
data "external" "generate-servicebus-sas" {
  program = ["Powershell.exe", "Set-ExecutionPolicy Bypass -Scope Process -Force; ./GenerateServiceBusSAS.ps1"]

  query = {
    servicebusUri = "${azurerm_servicebus_namespace.sb.name}.servicebus.windows.net"
    sbName        = azurerm_servicebus_namespace.sb.name
    policyName    = azurerm_servicebus_namespace_authorization_rule.authRule.name
    policyKey     = azurerm_servicebus_namespace_authorization_rule.authRule.primary_key
    sasExpiresInSeconds = 5256000
  }
}

The Powershell script invoked is a slightly adjusted version of the one from the Microsoft docs. We generate the SAS token on service bus level, not on topic level. The interesting part regarding Terraform and executing Powershell, is where the input parameters are read, and the output parameters are set.

After the ‘external’ resource has been ‘created’, the SAS token can be found in data.external.generate-servicebus-sas.result.sas

Import into API management

Once we have the service bus resources created, we can import the endpoint into API management. The script assumes there already is an APIM instance available, so it will not be created and needs to be configured in the apimResourceGroupName and apimName local variables. As service bus doesn't expose an OpenAPI document, we need to hand-craft and embed it in Terraform. In this example we add read (GET) and write (POST) operations on two topics to one API in APIM.

# Add endpoints to APIM
resource "azurerm_api_management_api" "apiEndpoint" {
  name                = "${azurerm_servicebus_namespace.sb.name}-sb"
  resource_group_name = data.azurerm_api_management.apim.resource_group_name
  api_management_name = data.azurerm_api_management.apim.name
  display_name        = "messagebus"
  revision            = 1
  path                = "messagebus"
  service_url         = "https://${azurerm_servicebus_namespace.sb.name}.servicebus.windows.net"
  protocols           = ["https"]

  import {
    content_format = "openapi+json"
    content_value  = <<JSON
    {
      "openapi": "3.0.1",
      "info": {
          "title": "apim to servicebus",
          "description": "",
          "version": "1.0"
      },
      "servers": [
          {
              "url": "${data.azurerm_api_management.apim.gateway_url}/messagebus"
          }
      ],
      "paths": {
          "/orders": {
              "get": {
                  "summary": "from ${azurerm_servicebus_subscription.sbOrdersSubscription.name} subscription",
                  "operationId": "${azurerm_servicebus_subscription.sbOrdersSubscription.name}-subscription",
                  "responses": {
                      "200": {
                          "description": null
                      }
                  }
              },
              "post": {
                  "summary": "to ${azurerm_servicebus_topic.sbOrdersTopic.name}",
                  "operationId": "post-to-${azurerm_servicebus_topic.sbOrdersTopic.name}",
                  "responses": {
                      "200": {
                          "description": null
                      }
                  }
              }
          },
          "/customers": {
              "get": {
                  "summary": "from ${azurerm_servicebus_subscription.sbCustomersSubscription.name} subscription",
                  "operationId": "${azurerm_servicebus_subscription.sbCustomersSubscription.name}-subscription",
                  "responses": {
                      "200": {
                          "description": null
                      }
                  }
              },
              "post": {
                  "summary": "to ${azurerm_servicebus_topic.sbCustomersTopic.name}",
                  "operationId": "post-to-${azurerm_servicebus_topic.sbCustomersTopic.name}",
                  "responses": {
                      "200": {
                          "description": null
                      }
                  }
              }
          }
      },
      "components": {
          "securitySchemes": {
              "apiKeyHeader": {
                  "type": "apiKey",
                  "name": "Ocp-Apim-Subscription-Key",
                  "in": "header"
              },
              "apiKeyQuery": {
                  "type": "apiKey",
                  "name": "subscription-key",
                  "in": "query"
              }
          }
      },
      "security": [
          {
              "apiKeyHeader": []
          },
          {
              "apiKeyQuery": []
          }
      ]
    }
    JSON
  }
}

To make sure the call to the service bus endpoint contains the SAS token, we define a policy that takes care of applying it.

# Apply authentication policy to endpoints
resource "azurerm_api_management_api_policy" "apiPolicy" {
  api_name            = azurerm_api_management_api.apiEndpoint.name
  api_management_name = data.azurerm_api_management.apim.name
  resource_group_name = local.apimResourceGroupName

  xml_content = <<XML
    <policies>
        <inbound>
            <base />
            <set-header name="Content-Type" exists-action="override">
                <value>application/atom+xml;type=entry;charset=utf-8</value>
            </set-header>
            <set-header name="Authorization" exists-action="override">
                <value>@((string)"${data.external.generate-servicebus-sas.result.sas}")</value>
            </set-header>
        </inbound>
        <backend>
            <forward-request />
        </backend>
        <outbound>
            <base />
        </outbound>
        <on-error>
            <base />
        </on-error>
    </policies>
  XML
}

Finally, we have to rewrite the URL to the service bus resource, as the url suffix we use is not the one service bus expects. We expose the endpoint via APIM as <apim>/messagebus/orders for orders, but need to rewrite this to <service bus>/orders/messages so it ends up at the correct topic. The rewrite is defined as operation level policy.

# Apply url rewrite policy to operations, to forward the message to the correct topic
resource "azurerm_api_management_api_operation_policy" "operationPolicyOrdersPost" {
  api_name            = azurerm_api_management_api.apiEndpoint.name
  api_management_name = data.azurerm_api_management.apim.name
  resource_group_name = local.apimResourceGroupName
  operation_id        = "post-to-${azurerm_servicebus_topic.sbOrdersTopic.name}"

  xml_content = <<XML
    <policies>
        <inbound>
            <base />
            <rewrite-uri template="/${azurerm_servicebus_topic.sbOrdersTopic.name}/messages" />
        </inbound>
        <backend>
            <forward-request />
        </backend>
        <outbound>
            <base />
        </outbound>
        <on-error>
            <base />
        </on-error>
    </policies>
  XML
}
resource "azurerm_api_management_api_operation_policy" "operationPolicyCustomersPost" {
  api_name            = azurerm_api_management_api.apiEndpoint.name
  api_management_name = data.azurerm_api_management.apim.name
  resource_group_name = local.apimResourceGroupName
  operation_id        = "post-to-${azurerm_servicebus_topic.sbCustomersTopic.name}"

  xml_content = <<XML
    <policies>
        <inbound>
            <base />
            <rewrite-uri template="/${azurerm_servicebus_topic.sbCustomersTopic.name}/messages" />
        </inbound>
        <backend>
            <forward-request />
        </backend>
        <outbound>
            <base />
        </outbound>
        <on-error>
            <base />
        </on-error>
    </policies>
  XML
}

We do the same for the read-and-delete (destructive read) operations on the API, where the URL needs to be constructed to contain the topic and subscription. When you need to perform a peek-lock operation instead, it's a matter of changing the HTTP verb to POST. Of course you manually would need to delete the message after reading and processing, in case of a peek-lock.

resource "azurerm_api_management_api_operation_policy" "operationPolicyOrdersGet" {
  api_name            = azurerm_api_management_api.apiEndpoint.name
  api_management_name = data.azurerm_api_management.apim.name
  resource_group_name = local.apimResourceGroupName
  operation_id        = "${azurerm_servicebus_subscription.sbOrdersSubscription.name}-subscription"

  xml_content = <<XML
    <policies>
        <inbound>
            <base />
            <set-method>DELETE</set-method>
            <rewrite-uri template="/${azurerm_servicebus_topic.sbOrdersTopic.name}/subscriptions/${azurerm_servicebus_subscription.sbOrdersSubscription.name}/messages/head" />
        </inbound>
        <backend>
            <forward-request />
        </backend>
        <outbound>
            <base />
        </outbound>
        <on-error>
            <base />
        </on-error>
    </policies>
  XML
}
resource "azurerm_api_management_api_operation_policy" "operationPolicyCustomersGet" {
  api_name            = azurerm_api_management_api.apiEndpoint.name
  api_management_name = data.azurerm_api_management.apim.name
  resource_group_name = local.apimResourceGroupName
  operation_id        = "${azurerm_servicebus_subscription.sbCustomersSubscription.name}-subscription"

  xml_content = <<XML
    <policies>
        <inbound>
            <base />
            <set-method>DELETE</set-method>
            <rewrite-uri template="/${azurerm_servicebus_topic.sbCustomersTopic.name}/subscriptions/${azurerm_servicebus_subscription.sbCustomersSubscription.name}/messages/head" />
        </inbound>
        <backend>
            <forward-request />
        </backend>
        <outbound>
            <base />
        </outbound>
        <on-error>
            <base />
        </on-error>
    </policies>
  XML
}

This blog post is about exposing the service bus REST endpoint in API management, so I won't go into the details of securing the public endpoint itself. There is an excellent video about this topic to be found here by Toon Vanhoutte.

Testing

For testing we can use the Azure portal or a REST client like Nightingale or Thunder Client. My tool of choice however is Postman.

We can send a request to the endpoint, only providing the fact we want to send something to the ‘customers’ topic on the message bus.

curl --location --request POST 'https://didago-core-apim-dev-we.azure-api.net/messagebus/customers' \
--header 'Ocp-Apim-Subscription-Key: my-key' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "Hello from Postman"
}'

The response is HTTP 201, to indicate the message was successfully posted. This can be verified with Servicebus Explorer or the Azure portal. All details around accessing the topic are abstracted way so clients can easily send messages to the message bus.

After posting a message, some process also will read it from the subscription. In this case we only have to change the HTTP verb to GET, to retrieve the message.

curl --location --request GET 'https://didago-core-apim-dev-we.azure-api.net/messagebus/customers' \
--header 'Ocp-Apim-Subscription-Key: my-key'

In the response we'll find obviously the message body posted, but also all HTTP headers from the subscription. From a security point of view it is not a best practice to expose internal details, so you can use APIM to remove these headers before returning data to the client.

One thing that might be of interest is the BrokerProperties header, which contains message metadata.

{"DeliveryCount":1,"EnqueuedSequenceNumber":1,"EnqueuedTimeUtc":"Mon, 30 Aug 2021 21:01:53 GMT","MessageId":"f87cc164304b41f9821fc13bc6c14910","PartitionKey":"134","SequenceNumber":67835469387268097,"State":"Active","TimeToLive":922337203685.47754}

When you're not using a destructive read but peek-lock instead, you need to have the MessageId and LockId (which is not present in the example above, as it was read using a destructive read) to delete the message from the topic after processing.

Final thoughts

As you can see it's quite easy to, next to the regular API endpoints, also centralize management of service bus can be exposed via APIM. The real value is in the consistent and simplified way customers use endpoints within your organization, and the fact you have full control over who is interacting with your service bus. Hiding security implementation details from customers allows you independently change these details, or even replace the mechanism entirely.

In that sense it would be much better to be able to use managed identies for accessing service bus from APIM. Unfortunately, that is currently not supported yet and this means you have to work with keys and SAS tokens. It's a best practice to regularly rotate the keys and refresh the SAS token, so that's also something you need to keep mind with this solution. Fortunately, the Terraform deployment refreshes the SAS token for you, so you only need to redeploy.

The only real concern with having all endpoints in APIM, is the fact that it introduces a single point of access. When that point is breached or insufficiently protected, your endpoints might be exposed to the public with all possible consequences. Therefore it's important to have and regularly review a solid approach for securing the endpoints in APIM. Securing endpoints is one, but just as important is to monitor and audit what's going on within APIM, so you're alerted when suspicious behavior is found.

If you have any comments or remarks, you can reach me on Twitter @jeanpaulsmit.