Cloud Development: Integrating Azure App Service with an Event Hub
Code and essential configurations
This tutorial demonstrates the integration of a web application hosted on the Azure App Service platform with a data streaming service, the Azure Event Hubs. We will discuss a few Azure services configurations that can streamline cloud software development.
As we progress through the text, links to the official Azure documentation will be provided for reference.
“Simplicity is a great virtue but it requires hard work to achieve it and education to appreciate it. And to make matters worse: complexity sells better.”
Edsger W. DijkstraSimplicity — the art of maximizing the amount of work not done — is essential. Principle 10 of the Agile Manifesto
The problem to be solved
We are required to implement a web application running in the Azure App Service platform. On receiving a request from a user, the app is going to send a simple message to a data streaming service, the Azure Event Hubs.
The code for the web application should be identical in both local development and production environments, and no secrets to enable access to the Event Hub (EvH) will be stored or managed.
We will refer to the environment where our web application runs in the Azure cloud platform as the production environment.
The described scenario is similar to this:
Before we begin…
We will use Python to implement our web application, but any Azure-supported programming language can be used, along with the appropriate libraries dependencies.
In a typical scenario, we wouldn’t access the same EvH from both local development and production environments.
The approach described in the following sections for integrating a web application with an EvH may also apply to integrating it with other Azure services, such as Blob Storage or Key Vault.
Prerequisites for this tutorial
- An Azure subscription and basic knowledge on Azure services;
- A VS Code environment for Python 3.10+ with
flask,azure-eventhubandazure-identitydependencies installed. This is our local development environment; - An Azure App Service Plan in which we will create a Web App to represent the application running in production. We will refer to our web application running on Azure App Service platform as Web App.
- An Azure Event Hubs Namespace, which we will call
ev-test-ns, and an Event Hub (EvH) namedev-test.
In (very) short, an Azure App Service plan is a logical container for one or more web apps, that share the same resources, configurations and capabilities. An Azure Event Hubs Namespace, on the other hand, is a logical container for one or more Event Hubs.
The Web Application
In VS Code, we have a project with just two files: application.py and requirements.txt. Our requirements file content looks like this:
azure-eventhub==5.11.1
azure-identity==1.12.0
Flask==2.2.2The code for our web application is contained in the file application.py. This is a Flask application that has one endpoint located at the / path. When the Flask service is called, it creates an EvH producer using the EventHubProducerClient class and sends a message.
import os
from azure.eventhub import EventHubProducerClient
from azure.eventhub import EventData
from azure.identity import DefaultAzureCredential
from flask import Flask
app = Flask(__name__)
@app.route("/")
def index():
print("Yay! A new request!")
producer = EventHubProducerClient(
fully_qualified_namespace=os.environ.get("AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACE"),
eventhub_name="ev-test",
credential=DefaultAzureCredential(),
)
with producer:
event_data_batch = producer.create_batch()
event_data_batch.add(EventData('{"our_message": "This is a message"}'))
producer.send_batch(event_data_batch)
print("Message sent")
return "Ok"
if __name__ == "__main__":
app.run()In the EventHubProducerClient constructor, we provide the Event Hubs Namespace (ev-test-ns.servicebus.windows.net) as the fully_qualified_namespace argument, and the EvH name as the eventhub_name argument. In the credential argument, we instantiated the DefaultAzureCredential class, which provides a convenient way to authenticate with Azure services, as it goes through a sequence of credentials, looking for a valid one, based on the environment the application is running in. However, it is important to note that the DefaultAzureCredential class may not always meet the security requirements for production environments.
We will get the following error when we run our Python code in VS Code:
At this point we did not provide any credential in our local development environment.
A credential for the local development environment
In our local development environment, we will use the Azure CLI az login command to login to Azure. As soon as we hit the Enter key, a browser window opens as shown below.
We logged in using the Azure user account developer@great-organization.com. The following is the output of the az login command in the terminal:
$ az login
A web browser has been opened at https://login.microsoftonline.com/organizations/oauth2/v2.0/authorize.
Please continue the login in the web browser.
If no web browser is available or if the web browser fails to open, use device code flow with `az login --use-device-code`.
[
{
"cloudName": "AzureCloud",
"homeTenantId": "aedad54e-8457-413f-a21e-025b6c586a62",
"id": "22a642a4-02f9-42dc-99c6-75f9a04e5ca6",
"isDefault": true,
"managedByTenants": [
{
"tenantId": "d40a0563-270a-421b-9eab-f7274f901224"
}
],
"name": "Our Great Organization",
"state": "Enabled",
"tenantId": "aedad54e-8457-413f-a21e-025b6c586a62",
"user": {
"name": "developer@great-organization.com",
"type": "user"
}
}
]The DefaultAzureCredential class uses the credential logged into the Azure CLI, so in this case, the user developer@great-organization.com must have permission to send messages from the local development environment to the EvH.
When we try to run our web application again, this is the result:
That’s expected! Now it’s time to set up our EvH and grant permission to the user developer@great-organization.com to send messages.
The Event Hub setup
There are two things to check in order to ensure our web application can access the EvH: the networking configuration and the role assigned to our user developer@great-organization.com.
The Event Hubs Namespace networking configuration
Our Event Hubs Namespace, ev-test-ns, currently allows access from All networks, as shown in the Azure Portal. While a role permission is required to send or consume messages from the ev-test EvH, we should not expose our Azure resource to the internet.
In the last section we will address this issue. For now, our local web application should be able to reach the EvH.
The Event Hub authorization for the local development environment
In the Azure Portal, we can verify the available built-in roles in our EvH. We need the Azure Event Hubs Data Sender role, which grants the assigned identities the ability to send messages to the EvH.
Let’s assign the role to the user developer@great-organization.com using the Azure CLI. The scope option in the command below specifies the ev-test EvH.
$ az role assignment create --assignee "developer@great-organization.com" \
--role "Azure Event Hubs Data Sender" \
--scope "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test"
{
"canDelegate": null,
"condition": null,
"conditionVersion": null,
"description": null,
"id": "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test/providers/Microsoft.Authorization/roleAssignments/88888888-cbe0-9999-949f-123459ddf1b6",
"name": "13123112-4444-5555-6666-025b6c586a62",
"principalId": "44444888-5555-6666-6666-025b6c586a62",
"principalType": "User",
"resourceGroup": "myRG",
"roleDefinitionId": "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/providers/Microsoft.Authorization/roleDefinitions/0690f733-e34b-4d1b-96d2-87fd4223263d",
"scope": "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test",
"type": "Microsoft.Authorization/roleAssignments"
}That’s it! Now we can send a message from our local environment to the ev-test EvH.
Note the
--scopeparameter in theaz role assignment createcommand. It is useful to understand the scope format, which can be found in the Azure CLI oficial documentation.
In Azure Portal, we see that ev-test EvH have just received one message.
We have our web application working on local development environment. Let’s move on and deploy it to the Azure App Service platform.
Deploying to the Azure App Service
Given that we already have an App Service Plan (with Linux Operating System) named greatapps in place, let’s create a Web App called testgreatapp and deploy our Flask application in it.
$ az webapp create \
--resource-group myRG \
--plan greatapps \
--name testgreatapp \
--runtime "PYTHON:3.10"The command output provides details about our newly created Web App. It is also possible to create an App Service Plan and a Web App all at once with the az webapp up command.
In the next step, we set the build automation setting in the Web App environment.
$ az webapp config appsettings set \
--resource-group myRG \
--name testgreatapp \
--settings SCM_DO_BUILD_DURING_DEPLOYMENT=trueWith the build automation enabled in our Web App, we can now deploy a simple .zip file that contains the two files of our web application:
application.py: The Python code file. It is important to keep the file name as is, as the App Service platform uses it to detect the presence of a Flask application;requirements.txt: The web application dependencies. The App Service platform will automatically include the gunicorn HTTP Server to run our service on port 80.
To deploy our .zip package we use the az webapp deploy command:
$ az webapp deploy \
--resource-group myRG \
--name testgreatapp \
--src-path testgreatapp.zipTo monitor the Web App logs in a terminal:
$ az webapp log tail --resource-group myRG --name testgreatappNow, when trying to access https://testgreatapp.azurewebsites.net/ in the browser, we get a big Internal Server Error page, because the EvH configurations expected by the Web App are still missing in the Azure environment.
The application
testgreatappwas created just as a demonstration in this tutorial and it has already been deleted. After deleting it, the address https://testgreatapp.azurewebsites.net/ can be used again by anyone inside the Azure App Service platform.
Authorizing the Web App for Event Hub access
There are a few more things to configure:
- The web application expects to use the environment variable
AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACEas the Event Hubs Namespaceev-test-ns.servicebus.windows.net; - An Azure identity is needed for our Web App. Previously, in the local development environment, we logged into Azure with the
developer@great-organization.comuser; - Assign the role
Azure Event Hubs Data Senderto our Web App identity.
Fortunately, Azure App Service provides a convenient way to accomplish all the 3 steps listed above, which is the Service Connector under the Settings section in the Azure Portal.
We will create a service connection in Azure Portal, as shown below:
Azure Event Hubs Data Sender´.
Now we have created an EvH connection for the Web App. The service connector has created a System-assigned Managed Identity for the application.
After creating the service connection, we can see that the AZURE_EVENTHUB_FULLYQUALIFIEDNAMESPACE environment variable was created for us.
The Service Connector is a great tool to manage our integrations to other Azure services and it supports several target services.
Now we test our application again.
Several requests received, several messages sent! 🙌
Wrapping up
Recall The problem to be solved section and check what we’ve accomplished:
- We have integrated the web application to an Event Hub successfully;
- The same code runs on both local development and production (Azure cloud platform) environments;
- No secret management was required.
But wait… Some issues were left behind
Security, Networking and Cloud Resources Management are broad and complex topics. In this section, we will just scratch the surface of some issues related to the development of our testgreatapp application. The issues will be addressed by simple solutions that may not apply to many enterprise level deployments.
Restricting the Event Hub scope access
In the Authorizing the Web App for Event Hub Access section, it was shown that the Service Connector provided the Azure Event Hubs Data Sender built-in role assignment to our application’s system-assigned managed identity. However, the assignment was made at the Event Hubs Namespace level, rather than at the individual Event Hub level, giving the Web App access to all Event Hubs within the namespace.
At the time of writing, we cannot define an individual Event Hub scope in the Service Connector. So, we will fix it in three steps using Azure CLI:
- Get the Web App principal ID with command az resource show;
- Create a new role assignment for the Web App at the individual Event Hub level.
- Delete the Web App role assignment at the Event Hubs Namespace level;
$ az resource show --resource-group myRG \
--resource-type Microsoft.Web/sites \
--name testgreatapp \
--query identity.principalId --output tsv
a93bba8a-bc89-45e4-93f6-fb660a3d59c8
$ az role assignment create \
--assignee a93bba8a-bc89-45e4-93f6-fb660a3d59c8 \
--role "Azure Event Hubs Data Sender" \
--scope "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test"
$ az role assignment delete \
--assignee a93bba8a-bc89-45e4-93f6-fb660a3d59c8 \
--role "Azure Event Hubs Data Sender" \
--scope "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns"Compare the scopes in the az role assignment operations above.
With that, we have applied the Principle of Least Privilege to our Web App’s access to the EvH, granting it only the necessary permissions to the individual EvH.
Our Web App still works!
Using AAD groups to manage a development team access to Azure resources
In The Event Hub authorization for the local development environment section, the developer@great-organization.com user was assigned the built-in role Azure Event Hubs Data Sender in the EvH to enable access from the local development environment.
However, in a development team, managing the access of many software engineers to various Azure resources can become cumbersome. As a best practice, it is more efficient to create a group in Azure Active Directory (AAD), add users to the group, and assign roles to the group. This reduces the burden of managing the access of multiple engineers to Azure resources, making it easier to manage and update access control policies in a centralized manner.
First, we create a group in AAD using the Azure CLI (or it can also be done through the Azure Portal) with the command az ad group create.
$ az ad group create --display-name testgroup --mail-nickname developerThe command az ad group member add assigns the user account to the group, but it requires the --member-id parameter, which can be retrieved using the az ad user list command. The membership of the user in the group can then be verified using the az ad group member check command.
$ az ad user list \
--query "[?userPrincipalName=='developer@great-organization.com'].id" \
--output tsv
a0012312-4444–5555-6666-876543bdbaa4
$ az ad group member add --group testgroup \
--member-id a0012312-4444–5555-6666-876543bdbaa4
$ az ad group member check --group testgroup \
--member-id a0012312-4444–5555-6666-876543bdbaa4
{
"value": true
}The az ad group member list command lists the users of a group:
$ az ad group member list --group testgroup
[
{
"@odata.type": "#microsoft.graph.user",
"businessPhones": [],
"displayName": "John Doe",
"givenName": "User",
"id": "a0012312-4444–5555-6666-876543bdbaa4",
...
},
...
]Finally, the Azure Event Hubs Data Sender role can be “transferred” from the developer@great-organization.com user to the testgroup group. We will need the testgroup group’s Id, which can be obtained by using the az ad group show command.
$ az role assignment delete --assignee "developer@great-organization.com" \
--role "Azure Event Hubs Data Sender" \
--scope "/subscriptions/aedad54e-8457–413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test"
$ az ad group show --group testgroup --query id --output tsv
ea47439b-5ac3-497a-928f-7c7eadcda0f7
$ az role assignment create --assignee ea47439b-5ac3-497a-928f-7c7eadcda0f7 \
--role "Azure Event Hubs Data Sender" \
--scope "/subscriptions/aedad54e-8457-413f-a21e-025b6c586a62/resourcegroups/myRG/providers/Microsoft.EventHub/namespaces/ev-test-ns/eventhubs/ev-test"We have created an AAD group to which we will add the software engineers of our development team.
Our Web App still works!
Securing Event Hub Access: Preventing Internet Exposure
In The Event Hubs Namespace networking configuration section we left the Event Hubs Namespace accessible from all networks.
We can make it better with the option Selected networks. Please, refer to the oficial documentation to learn more about the access to Event Hubs Namespaces from Virtual Networks (VNet). For better security, however, we will disable the public network access and create a Private Endpoint Connection.
To use a Private Endpoint Connection in our scenario, we need a configured VNet and an Azure VPN Gateway service, which is necessary to access the EvH from the local development environment.
As expected, after disabling the public network access, our web application in both local development e production environments stopped working.
Then we create a Private endpoint.
The Create a private endpoint wizard in Azure Portal has quite a few options and requires that we already have a VNet in place. We filled in the basic information, then chose an existing VNet named MyNet in the Virtual Network tab.
We chose to integrate the private endpoint with a Private DNS Zone.
Now that we have created the private endpoint for the ev-test-ns Event Hubs Namespace, we should integrate the VNet MyNet into our Web App for the network outbound traffic. This means that the Web App is going to call the EvH through MyNet.
After selecting the MyNet VNet for the network outbound traffic, our Web App is working again!
However, the EvH is not accessible from the web application in the local development environment. This is where the Azure VPN Gateway comes into play. For our scenario, the VPN service should allow traffic over the VNet MyNet. This is the same VNet we integrated to the Web App (for outbound traffic) and from which the Private endpoint obtains its IP address.
Once we have an Azure VPN client set up and established a VPN tunnel from our local development environment (point-to-site VPN connection), we can successfully access the EvH from the local web application. It is working again!!! 🥳
One thing to note in the local environment configuration is the traffic routing through the VPN tunnel. For our scenario, the traffic to the DNS suffix .servicebus.windows.net should be routed through the VPN tunnel.
We have successfully created a Private endpoint for our Event Hub and made some configurations in order to access it from both local development and production environments. A VNet and an Azure VPN Gateway were required in our setup.
Our Web App still works!
It’s important to keep in mind that the software solutions we developed, configured, and deployed for our web application are simply a minimal representation of what’s possible. These configurations should not be taken as universally applicable or considered enterprise-grade, as the software ecosystem and enterprise policies surrounding each decision and configuration are vast.
That’s it! I hope this tutorial helps you on implementing better software.
Cloud software development can quickly become complex with the addition of various tools and technologies. To keep things manageable, it’s a good idea to start with a simple setup and only incorporate new tools as they are truly necessary. As Robert C. Martin (Uncle Bob) stated in the book Clean Architecture: “A good software architecture allows decisions about frameworks, databases, web servers, and other environmental issues and tools to be deferred and delayed.”
