Adventures with Azure: Cosmos DB, Azure Functions, Event Grid, Oh My!

To get a better understanding of the Cosmos DB Change, Event Grid, retry and fail over logic, I decided to build a system that would respond to changes in a Cosmos DB, send events to an Azure Event Grid, which in turn would forward the events to subscribers.

I broke this article in to three parts.

In Part 1, we will create a Cosmos DB Trigger Azure Function to respond to document adds and updates from a Cosmos DB database, which will then forward the documents to an Event Grid Topic.

We will create a single subscription to an Event Grid Topic which will forward events to an Azure Store Queue with an Event Grid Subscription.

If for some reason events can not be published to the Event Grid Topic, latency or throttling with Event Grid, the Cosmos DB Trigger Azure Function will make use of retry logic to attempt to publish the events again, until the retries are exhausted.

In Part 2, we will add an Event Grid Trigger Azure Function to subscribe to Event Grid messages.

In Part 3, we will add fail over logic to the Cosmos DB Trigger Azure Function if it is unable to publish event to the Event Grid topic , e.g. Event Grid is down, it will instead push the events to Azure Blob Storage.

Since Azure Functions do not support output bindings for Azure Event Grid We will have to publish the events to Azure Event Grid with code, using the Azure Event Grid SDK.

This article assumes some familiarity with Azure and creating various Azure Resources.

Part 1

Setup Azure Environment

Create the following Azure resources:

• Create a Resource Group
• Create a Storage Account
• Create a Function App
  • Create with an Application Insights resource
• Create an Event Grid Topic
• Create a Storage Queue for the Storage Account
  • Name the storage queue the same as the Event Grid Topic created in the previous step
• Create an Event Grid Subscription
  • Name the subscription the same as the Storage Queue created in the previous step
  • Create with a Storage Queue endpoint
• Create a Cosmos DB account
  • Create a database for the Cosmos DB account
  • Create a container for the Cosmos DB database, called persons

Create the Function App Project

Open up Visual Studio, or VS Code.

Create an Azure Function App project, I named mine FuncApp1.

Add a folder named Helpers.

Add a class named Person.cs.

Paste into your class the code snippet as follows:

These are the documents that we will write to the Cosmos DB database and send as events to Azure Event Grid.

Edit the file local.settings.json as follows:

Replace the values that start with YOUR with the values from your Azure resources.

Create the Http Trigger to Populate Cosmos DB Database

Add an Http Trigger called HttpTrigger1.cs.

Paste into your class the code snippet as follows:

Install the NuGet package Bogus.

This function will insert 10 person documents into the Cosmos DB database that will trigger the Cosmos DB Trigger, which we will create in the next step.

Create the Cosmos DB Trigger to Read from Cosmos DB Change Feed

Add a Cosmos DB Trigger called CosmosDBTrigger1.cs and paste the code snippet into the call as follows:

This Azure Function will respond to any adds or updates from the persons container in Cosmos DB.

Install the NuGet package Microsoft.Azure.EventGrid.

We will want to provide our own HttpClient as opposed to letting the EventGridClient manage it for us, if we don’t, we could run into Socket Exceptions.

See Managing Connections for more information.

Since Event Grid does not support retry logic out of the box, we add support ourselves using Polly.

Install the NuGet package Polly.

Add another class, HttpRetryMessageHandler.cs, to the Helpers folder and paste the code snippet into the class as follows:

This class creates a retry policy with an exponential back-off.

• After the first failure the code will wait 3 seconds before it tries to execute again.
• After the second failure the code will wait 9 seconds.
• After the third failure it will wait 27 seconds.
• After the fourth failure it will throw an Exception.

Run the Azure Function project.

Call the Http Trigger Azure Function, using Postman, with a POST request to http://localhost:7071/api/HttpTrigger1 to insert 10 person documents into Cosmos DB.

Let’s create the Event Grid Subscription.

Navigate to the Event Grid Topic and click Add Subscription.

Enter a name.

Select Storage Queues for Endpoint Type.

Click Select an endpoint.

Select the Storage Account and Storage Queue that was created earlier.

Click Create.

Call the Http Trigger Azure Function to insert 10 person documents into Cosmos DB.

Open up Azure Storage Explorer.

Navigate to the Storage Account we created earlier, expand Queues and click on the Storage Queue.

You should see a list of the Cosmos DB documents that were sent to Event Grid and forwarded to the Storage Queue subscription.

One thing to note, our Cosmos DB Trigger Azure Function is sending events to Azure Event Grid in batches, as Azure Event Grid expects an array of events.

When our events reach Azure Event Grid, they are debatched and sent to our endpoints individually.

Part 2

Now let’s add an Azure Function to receive events from our Event Grid Topic.

Add a Event Grid Trigger called EventGridTrigger1.cs.

Paste the code snippet below into the EventGridTrigger1.cs class as follows:

Since it is difficult to test the Event Grid Azure Function locally, let’s publish the Function App to Azure.

Make sure to include your Application Settings by clicking Edit Azure App Service Settings.

In the Azure portal, navigate to the Function App and select the EventGridTrigger1 function.

Click the Add Event Grid subscription link.

Provide a name for the subscription.

Select Event Grid Topic for the Topic Type.

Select the subscription, resource group, and finally the Azure Event Grid Topic.

Click Create.

If you navigate to your Event Grid Topic you will now see two Event Grid Subscription, the first to our Azure Storage Queue and the second to our Event Grid Trigger Azure Function.

Before we test this locally, we will need to disable the CosmosDBTrigger1 from the portal, if we don’t, there is a chance, it may respond to our Cosmos DB changes before our local environment can.

Run the Azure Function project.

Call the Http Trigger Azure Function to insert 10 person documents into Cosmos DB.

Navigate back to the Azure Function App, select the EventGridTrigger1 and then select the Monitor node.

You should see a list of the events received from Azure Event Grid.

Part 3

So what happens if we are unable to send an event to Azure Event Grid? Remember our retry policy will try three times and on the final attempt it will throw an Exception.

If we cannot send an event to Azure Event Grid we will write the event to Azure Blob Storage so it can be processes later, maybe by a Time Trigger Azure Function.

We will need to update the CosmosDBTrigger1 with a couple of code changes.

Update the Run method to include a parameter for the Blob output binding, I added mine right after ILogger logger.

We will also want to wrap the eventGridClient.PublishEventSync() method in a try...catch, and in the catch add the code to write the event to Azure Blob Storage.

We will throw the Exception again so that Application Insights tracks the exception.

To simulate a failure to call Azure Event Grid update the value for AzureEventGrid:TopicEndpoint in local.settings.json to a bad value, I just add an additional character.

Create a Blob container called cosmsosdbtrigger-errors to capture the events.

Run the Azure Function project.

Call the Http Trigger Azure Function to insert 10 person documents into Cosmos DB.

Open Azure Storage Explorer.

Navigate to the Blob storage container cosmsosdbtrigger-errors.

You should see a list of the events sitting in Azure Blob Storage.

Apologize for the long article, but felt it best to pack it all together as I have been working better practices for retry logic for sending to Event Grid in an Azure Function, and fail over logic.

Feedback is much appreciated!