Protecting Azure Storage Account Queues via API Management

Sometimes you need to receive and process messages from a 3rd party supplier, but:

  • the supplier expects an HTTP endpoint to send the messages to
  • you want to decouple receiving from processing, because you want asynchronous processing

In this scenario the standard approach would be to use a queue, that can be either an Azure Service bus queue or an Azure Storage Account queue. For this use case I'm using a Storage account queue. Next to the SDK, these queues can be accessed via a REST endpoint as well. This seems convenient as you can provide this URL directly to the supplier, but this also means you setup tight coupling between supplier and one of your resources. It's a best practice to keep control over your resources by means of indirection, as it provides you with the flexibility to intercept, secure and/or validate the messages before they end up in the queue. This layer of indirection, like you expect for all of your HTTP endpoints, is API Management.

Scenario

The scenario is rather simple:

Expose inbox endpoint via API management

We're going to:

  • expose an HTTP endpoint on API management
  • create a Storage account queue
  • protect the Storage account queue from unauthorized access, by means of Azure AD and network restrictions
  • use Terraform to configure all of this

The endpoint for the supplier in this scenario can be extended with Azure AD or other security mechanisms, but that's beyond the scope of this blog post. The goal is to protect the queue by putting API management in front of it.

Terraform

For this scenario I'm going to assume you already have an API management instance ready. If you want to add this to your provisioning, you can check you this previous blog post. Important here, is that your API management instance needs to have managed identities enabled. This can be found in the security section of the resource’ menu.

API management managed identity

Securing the storage account with access restriction to only API management, is unfortunately not supported in Terraform yet. We're going to use the AzApi provider for this.

Provisioning steps:

  • Get reference to existing API management resource
  • Create resource group
  • Create storage account
  • Create storage account queue
  • Assign Storage Queue Data Message Sender role to the API management resource principal (managed identity)
  • Close down network access to all traffic, except the API management resource
  • Add API, operation and policy to API management

The API management policy takes care of the technical things needed to access the queue. One of them is specifying the version of the REST API used. After all, the policy does nothing but a secure POST to the queue's endpoint.

This results in the following Terraform script, which can be found in this repo.

# Specify location to store tfstate files
terraform {
  required_providers {
    azurerm = {
      source = "hashicorp/azurerm"
      version = "= 3.15"
    }
    azapi = {
      source  = "Azure/azapi"
    }
  }
  required_version = "= 1.2.5"
}

provider "azurerm" {
  features {}
}
provider "azapi" {
}

# Reference to the environment
data "azurerm_client_config" "current" {}

# Reference to API Management
data "azurerm_api_management" "apim" {
  name                = "didago-apim"
  resource_group_name = "didago-apim-rg"
}

# Create resource group for the storage account
resource "azurerm_resource_group" "queue-rg" {
  name = "didago-queue-rg"
  location = "westeurope"
}

# Create storage account
resource "azurerm_storage_account" "sa" {
  name                     = "didagosaqueue"
  resource_group_name      = azurerm_resource_group.queue-rg.name
  location                 = azurerm_resource_group.queue-rg.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
}

# Create the queue
resource "azurerm_storage_queue" "queue" {
  name                 = "message-processing"
  storage_account_name = azurerm_storage_account.sa.name
}

# Assign Storage Queue Data Message Sender permissions to API management resource
resource "azurerm_role_assignment" "assign-send-permissions" {
  scope                = azurerm_storage_account.sa.id 
  role_definition_name = "Storage Queue Data Message Sender"
  principal_id         = data.azurerm_api_management.apim.identity[0].principal_id
}

# Deny access to any other resource than API management
resource "azapi_update_resource" "secure-sa" {
  type        = "Microsoft.Storage/storageAccounts@2021-09-01"
  resource_id = azurerm_storage_account.sa.id

  body = jsonencode({
    properties = {
      networkAcls = {
      defaultAction = "Deny"
      resourceAccessRules = [{
        resourceId = data.azurerm_api_management.apim.id
        tenantId   = data.azurerm_client_config.current.tenant_id
      }]
      }
    }
  })
}

# Create API in API management
resource "azurerm_api_management_api" "api" {
  name                = "queue-api"
  resource_group_name = data.azurerm_api_management.apim.resource_group_name
  api_management_name = data.azurerm_api_management.apim.name
  revision            = "1"
  display_name        = "Inbox API"
  path                = "inbox"
  protocols           = ["https"]
  service_url         = "${azurerm_storage_account.sa.primary_queue_endpoint}${azurerm_storage_queue.queue.name}"
}

# Add operation to API
resource "azurerm_api_management_api_operation" "operation" {
  operation_id        = "post-message"
  api_name            = azurerm_api_management_api.api.name
  api_management_name = azurerm_api_management_api.api.api_management_name
  resource_group_name = data.azurerm_api_management.apim.resource_group_name
  display_name        = "Post message to the inbox"
  method              = "POST"
  url_template        = "/"
  description         = "Post message to backend queue"

  response {
    status_code = 202
  }
}

# Configure policy on operation
resource "azurerm_api_management_api_operation_policy" "policy" {
  api_name            = azurerm_api_management_api_operation.operation.api_name
  api_management_name = azurerm_api_management_api_operation.operation.api_management_name
  resource_group_name = azurerm_api_management_api_operation.operation.resource_group_name
  operation_id        = azurerm_api_management_api_operation.operation.operation_id

  xml_content = <<XML
  <policies>
      <inbound>
          <base />
          <set-header name="x-ms-version" exists-action="override">
              <value>2021-08-06</value>
          </set-header>
          <authentication-managed-identity resource="https://storage.azure.com/" />
          <set-header name="content-type" exists-action="override">
              <value>application/xml</value>
          </set-header>
          <set-body>@{
                  return "<QueueMessage><MessageText>" + context.Request.Body.As<string>() + "</MessageText></QueueMessage>";
              }</set-body>
          <rewrite-uri template="/messages" copy-unmatched-params="true" />
      </inbound>
      <backend>
          <forward-request />
      </backend>
      <outbound>
          <return-response>
              <set-status code="202" />
          </return-response>
          <base />
      </outbound>
      <on-error>
          <base />
      </on-error>
  </policies>
XML
}

Due to the role assignment, API management can authenticate itself against the storage queue resource with Azure AD. The authentication-managed-identity policy part takes care of generating an Azure AD JWT and present that to the storage REST endpoint. To access the queue on the REST endpoint, the URL needs to have the /messages suffix, which is handled by the rewrite-uri section. An HTTP 202 (accepted) is returned to the supplier, which indicates the message is received and accepted, but processing is asynchronous. When needed, you could consider to return a custom URL, with which the caller then could retrieve status information on the progress of the processing.

Testing

For testing we can use the Azure portal or a REST client like Nightingale, Thunder Client or Postman. But did you know Visual Studio Code is also capable of doing so?

curl --location --request POST 'https://didago-apim.azure-api.net/inbox/' \
--header 'Ocp-Apim-Subscription-Key: b8f72cffdei0d7a1g4o7d014f77759117' \
--header 'Content-Type: application/json' \
--data-raw '{
    "message": "Hello from VS Code"
}'

VS Code REST client

The response is HTTP 202, to indicate the message was successfully posted.

If you would check out the queue in the Azure portal, or via some tool, you will find out you don't have access. This is due to the lock down of all access, except coming from API management. You can add your own IP address to by-pass this, although this is not a sustainable or secure solution.

Storage account network settings

Final thoughts

Exposing access to a queue in this way is easy and secure. There is no SAS or other token in the policy, like was the case in one of my previous blog posts. This also saves you from having to store or refresh such a token. Managed identities are the preferred way of authentication, when it concerns Azure-to-Azure resources.

If you have any comments or remarks, you can reach me on Twitter @jeanpaulsmit.