In a recent project I had dig deeper into how the Cosmos DB change feed worked, specifically in conjunction with the Azure Function Cosmos DB Trigger.
I had some assumptions on how the Cosmos DB change feed worked, but needed to validate them.
This will like be a series of articles.
Before we go any further, let’s set the stage …
For the sake of this article, I am assuming you are familiar with Cosmos DB, Azure Functions and deploying Azure Functions from Visual Studio.
The source code is located at
https://github.com/mattruma/SampleCosmosDbChangeFeed.
What is CosmosDB change feed support?
According to Microsoft Documents:
Change feed support in Azure Cosmos DB works by listening to an Azure Cosmos DB container for any changes.
It then outputs the sorted list of documents that were changed in the order in which they were modified.
The changes are persisted, can be processed asynchronously and incrementally, and the output can be distributed across one or more consumers for parallel processing.
https://docs.microsoft.com/en-us/azure/cosmos-db/change-feed
How does the Azure Function work with the Cosmos DB change feed?
The Azure Function Cosmos DB Trigger relies on the change feed support through the implementation of an additional collection, referred to as the lease container.
According to the Microsoft Documents:
The lease container coordinates processing the change feed across multiple workers.
A separate container is used to store the leases with one lease per partition.
It is advantageous to store this lease container on a different account with the write region closer to where the change feed processor is running.
https://github.com/MicrosoftDocs/azure-docs/blob/master/articles/cosmos-db/change-feed-processor.md
In simpler terms, the lease container holds a pointer to the current position in the change feed for a particular partition.
Let’s get to the code setup the environment in Azure.
You will need to Create an Azure Cosmos DB Account.
Once your Cosmos DB Account has been created, navigate to the Cosmos DB and click Quick start and then click Create “Items” collection.
This will create a database called ToDoList and a single collection called Items.
We will want to delete the current Items collection and recreate it with a partition key – the default Items collection created DOES NOT have a partition key.
Click on Data Explorer.
Right click on the Items collection and click Delete Container. This will delete the container from the database.
Right click on the ToDoList database and click Create Container.
- Set the Container id to
Items - Set the Partition key to
/id - Set the Throughput to
400, this is the cheapest collection you can have.
We now need to Create our Azure Function Apps.
We need to create five in one region, e.g. US East 2 and another in a different region, e.g. US West 2.
Download the source code at
https://github.com/mattruma/SampleCosmosDbChangeFeed and open it in Visual Studio.
You will want to modify the local.settings.json to look similar to the following:
|
1 2 3 4 5 6 7 8 9 10 |
{ "IsEncrypted": false, "Values": { "AzureWebJobsStorage": "UseDevelopmentStorage=true", "FUNCTIONS_WORKER_RUNTIME": "dotnet", "CosmosDB:DatabaseId": "Items", "CosmosDB:ConnectionString": "YOUR_COSMOS_DB_CONNECTION_STRING", "CosmosDB:MaxConcurrency": 6 } } |
Where the heck is my Cosmo DB Connection String?
Navigate to the Azure Cosmos DB Account, click on Keys and copy the Primary Connection String.
Do this for each of the Function Apps.
You will now want to deploy each of these Function Apps to the Function Apps you setup in Azure.
In my case I deployed my Visual Studio projects (to the left) to the Azure resources (to the right) as follows:
- FunctionApp1 →
YOUR_PREFIX-001-funcapp - FunctionApp2 →
YOUR_PREFIX-002-funcapp - FunctionApp3 →
YOUR_PREFIX-003-funcapp - FunctionApp4 →
YOUR_PREFIX-004-funcapp - FunctionApp5 →
YOUR_PREFIX-005-eastus-funcapp - FunctionApp5 →
YOUR_PREFIX-005-westus-funcapp
We deploy FunctionApp5 twice, once in one region, in my case, East US and another in another region, in my case West US.
You will need to update the Application Settings for each Function App in Azure.
If deploying from Visual Studio you can configure the Application Settings by clicking Edit Azure App Service Settings in the publish wizard.
That should set everything up!
In the next article in this series we will start going through each of the functions and this is the where we will being to understand the real magic this going on with the Azure Cosmos DB change feed.
Related Articles