Azure Cosmos DB

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Cosmos DB–Multi Master

October 8, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Azure, Azure CLI, Azure Cosmos DB, CosmosDB, Data Consistancy, Data Integrity, Microsoft, Multi-master, Performance, Reliability, Resilliancy, Scalability, Scale Up No comments

During the Ignite 2018, Microsoft has announced the general availability of Multi-Master feature being introduced to Azure Cosmos DB to provide more control into data redundancy and elastic scalability for your data from different regions with multiple writes and read instances.

What is Multi-Master essentially?

Multi-master is a capability that provided as part of Cosmos DB, that would provide you multiple write regions and provides an option to handle conflict resolution automatically through different options provided by the platform. Most of the major scenarios you would encounter the conflict can be resolved with these simple configurations.

A sample diagram depicting a use case of load balanced web app writing to respective regional master:-

image

With multi-master, Azure Cosmos DB delivers a single digit millisecond write latency at the 99th percentile anywhere in the world, and now offers 99.999 percent write availability (in addition to 99.999 percent read availability) backed by the industry-leading SLAs.

image

Wow! That’s an amazing performance Cosmos DB guarantees to provide so that your mission-critical systems will have zero downtime, if they start using Cosmos DB.

 

How to Enabled Multi-Master support in your Cosmos DB solutions?

Currently multi-master can only be enabled for new Cosmos DB instances using “Enable Multi-Master” option in Azure Portal or through PowerShell or ARM templates or through SDK.

These options are detailed below with necessary examples:

1.) Azure Portal – Enable Multi-region writes and Enable geo-redundancy

image

2.) Azure CLI 
Set the “enable-multiple-write-locations” parameter to “true”

az cosmosdb create \
   –-name "thingx-cosmosdb-dev" \
   --resource-group "consmosify-dev" \
   --default-consistency-level "Session" \
   --enable-automatic-failover "true" \
   --locations "EastUS=0" "WestUS=1" \
   --enable-multiple-write-locations true \

3.) AzureRM PowerShell
In AzureRM PowerShell cmdlet – Set enableMultipleWriteLocations parameter to “true”

$locations = @(@{"locationName"="East US"; "failoverPriority"=0},
             @{"locationName"="West US"; "failoverPriority"=1})

$iprangefilter = ""

$consistencyPolicy = @{"defaultConsistencyLevel"="Session";
                       "maxIntervalInSeconds"= "10";
                       "maxStalenessPrefix"="200"}

$CosmosDBProperties = @{"databaseAccountOfferType"="Standard";
                        "locations"=$locations;
                        "consistencyPolicy"=$consistencyPolicy;
                        "ipRangeFilter"=$iprangefilter;
                        "enableMultipleWriteLocations"="true"}

New-AzureRmResource -ResourceType "Microsoft.DocumentDb/databaseAccounts" `
  -ApiVersion "2015-04-08" `
  -ResourceGroupName "consmosify-dev" `
  -Location "East US" `
  -Name "thingx-cosmosdb-dev" `
  -Properties $CosmosDBProperties

4.) Through CosmosDB SDK
Setting connection policy in DocumentDBClient and set UseMultipleWriteLocations to true.

ConnectionPolicy policy = new ConnectionPolicy
{
   ConnectionMode = ConnectionMode.Direct,
   ConnectionProtocol = Protocol.Tcp,
   UseMultipleWriteLocations = true,
};
policy.PreferredLocations.Add("East US");
policy.PreferredLocations.Add("West US");
policy.PreferredLocations.Add("West Europe");
policy.PreferredLocations.Add("North Europe");
policy.PreferredLocations.Add("Southeast Asia");
policy.PreferredLocations.Add("Japan East");
policy.PreferredLocations.Add("Japan West");

Azure Cosmos DB multi-master configuration is the game changes that really makes it a true global scale database with automatic conflict resolution capabilities for data synchronization and consistancy.

In my later sessions I will write examples to cover how conflict resolutions can be configured and used in realtime scenarios.

Useful Refs:

Azure Cosmos DB–Reserved Capacity

October 7, 2018 Azure, Azure Cosmos DB, Billing, CosmosDB, Enterprise Agreement, Infrastructure, Microsoft, Pay-as-you-go No comments

Azure Cosmos DB is a planet scale global document database which have been available for Azure Customers based on pay-as-you-go. Reserved Capacity is a new long term pre-paid billing commitment customers can get a discounted pricing. Azure Cosmos DB reserved capacity helps you save money by pre-paying for Azure Cosmos DB resources for a period of one year or three years.

  • Reserve capacity allows you to get a optimum discount for the throughput(RUs) provisioned for Cosmos DB resources.
  • For Ex: Databases and Containers (tables/collections/graphs).
  • It can significantly reduce your Cosmos DB costs and it enables you to save up to 65 percent on regular prices with one-year or three-year upfront commitment.
  • Reserved capacity provides a billing discount and does not affect the runtime state of your Cosmos DB resources.
  • It is available to all supported APIs (including MongoDB, Cassandra, SQL, Gremlin and Azure Tables) and all regions worldwide.
  • Scope of Reservation:

    A reservation is scoped to a single subscription or shared across all your enrolment:

    • Scoped to a Single Subscription: that means a set of cosmos db account resources such as database or containers within the selected subscription.
    • Shared/Scoped to an enrolment: Billing benefit can be shared across any subscription in the enterprise agreement or pay-as-you-go subscriptions.

    cosmos-db-reserved-capacity

    Who can opt-in for Reserved Capacity?

    This feature is currently only applicable for enrolment (Enterprise Agreement customers) or account (Pay-as-you-go customers). Users with MSDN Subscription benefits are not applicable for the Reserved Capacity benefits.

    You can buy Azure Cosmos DB reserved capacity from the Azure portal.

    1. Sign in to the Azure portal.
    2. Select All services > Reservations > Add and Select Product Type > Cosmos DB

    image

    8fafbeff-f11c-4894-98e0-e03d79d1787d

    For more details review reserved capacity documentation.

    Azure Cosmos DB–Setting Up New Database using Azure CLI–Sample

    October 1, 2018 Azure, Azure Cosmos DB, Codes, CosmosDB, KnowledgeBase, Microsoft, PowerShell, Windows Azure Development No comments

    Purpose of this article is to help you with few steps of commands to provision a new Azure Cosmos DB database instance through Azure CLI or Azure Cloud Shell.

    Here is the snippet:

     <# 
       This Bash script should help you create a Azure Cosmos DB instance using Azure CLI with bare minimal configuration 
    #>
    
    export ACCOUNT_NAME="thingx-retail-store-db"
    export DB_RESOURCE_GROUP="thingx-dev"
    export DB_LOCATION="southcentralus"
    export DB_NAME="Products" 
    export DB_THROUGHPUT=1000  ## bare minimal for 500 read and 100 write configuration for a 1KB document.
    export DB_COLLECTION_NAME="Groceries"
    
    ##Optional: If resource group does not exist create a new one 
    az group create --name $DB_RESOURCE_GROUP --location $DB_LOCATION
    
    ##1.0 Create the Azure Cosmos DB Account 
    az cosmosdb create --name $ACCOUNT_NAME --kind GlobalDocumentDB --resource-group $DB_RESOURCE_GROUP
    
    ##2.0 Create Products database in the account  
    az cosmosdb database create --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP
    
    ##3.0 Create Groceries collection in Products database
    az cosmosdb collection create --collection-name $DB_COLLECTION_NAME --partition-key-path "/productId" --throughput $DB_THROUGHPUT --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP
    
    
    

     

    For the ease of this article I used Azure Cloud Shell, that you can launch from your azure portal by clicking on the Shell icon on the top portal menu.

    image

    image

    Now you are ready to execute all the commands listed above in the sample bash script

    image

    Create New Azure Cosmos DB Account

     <span data-mce-type="bookmark" style="display: inline-block; width: 0px; overflow: hidden; line-height: 0;" class="mce_SELRES_start"></span>
    az cosmosdb create --name $ACCOUNT_NAME --kind GlobalDocumentDB --resource-group $DB_RESOURCE_GROUP

    image

    Create Products Database in the account

    az cosmosdb database create --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

    image

    Create Groceries collection in Products database

    az cosmosdb collection create --collection-name $DB_COLLECTION_NAME --partition-key-path "/productId" --throughput $DB_THROUGHPUT --name $ACCOUNT_NAME --db-name $DB_NAME --resource-group $DB_RESOURCE_GROUP

    image

    Now if you browse Azure Portal you can see resources created in “thingx-dev” resource group.

    image

    Upon browsing with Data Explorer you can see the Groceries collection inside Products DB.

    image

    So that’s the few silly easy steps to create Cosmos DB database from Azure CLI or Azure Cloud Shell. Hope that makes it easy for you.

    70-777 – Azure Cosmos DB Certification Exam (BETA)

    August 18, 2018 .NET, Azure, Azure Cosmos DB, Certification, CosmosDB, MCP, Microsoft, Microsoft Specialist No comments

    70-777: Implementing Microsoft Azure Cosmos DB Solutions”  is the new Microsoft certification in the town for any Developer or Architect who leverages Azure Cosmos DB. If you understand how Cosmos DB database can be configured and consumed using platform agnostic Cosmos DB API, this certification will help you prove your expertise.

    To read more about the exam 70-777 visit: https://www.microsoft.com/en-us/learning/exam-70-777.aspx  ( the link will also provide you guidance on the topics covered by this exam)

    As on today, 70-777 BETA exam enrollment started, has around 300 beta seats for this exam, and you must register and take the exam before October,1 2018. Hurry if you are passionate enough to learn and try out this certification exam, please register now before you quote is full.

    Steps to Utilize this Offer:

    Source: Born to Learn Community

    Azure Cosmos DB – Change feed support(PREVIEW)–available

    July 26, 2018 .NET, .NET Core, .NET Framework, Azure, Azure Cosmos DB, C#.NET, CosmosDB, Microsoft No comments

    Today Microsoft announced the preview of Change feed Support for Azure Cosmos DB, which allows you to build scalable solutions. By default change feed will be enabled in all the accounts.

    Change feed provides an output of sorted list of documents that has been changed in the order in which they are modified by client operations. These changes are persisted, can be processed asynchronously and incrementally, they enables developers to write alternative logic to operate upon these change for generation reports, or invoke another operation such as sending email or audit logs etc.

    ChangeFeedProcessor

    Start using :

    Source: