Visual Studio 2017

Azure Cosmos DB – TTL (Time to Live) – Reference Usecase

October 9, 2018 .NET, .NET Core, .NET Framework, Analytics, Architecture, Azure, Azure, Azure Cosmos DB, Azure Functions, Azure IoT Suite, Cloud Computing, Cold Path Analytics, CosmosDB, Emerging Technologies, Hot Path Analytics, Intelligent Cloud, Intelligent Edge, IoT Edge, IoT Hub, Microsoft, Realtime Analytics, Visual Studio 2017, VisualStudio, VS2017, Windows No comments

TTL capability within Azure Cosmos DB is a live saver, as it would take necessary steps to purge redudent data based on the configurations you may. 

Let us think in terms of an Industrial IoT scenario, devices can produce vast amounts of telemetry information, logs and user session information that is only useful until we operate on them and take action on them, to be specific up to finate period of time. Once that data becomes surplus, we need an application logic that purges these old records.

With the “Time to Live” or TTL, Microsoft Cosmos DB provides an ability to have your documents automatically purged from database storage after a certian period if time(which you configured)

  • This TTL by default can be set on a document collection level and later can be overridden on a per document basis.
  • Once the TTL is set, Cosmos DB service will automatically remove the documents when its lifetime is over.
  • Inorder to track TTL, Cosmos DB uses an offset field to check when it was last modified.  This field is identifiable as “_ts”, which exists in every document you create.  Basically it is a UNIX epoch timestamp. This field is updated everytime when the document is modified. [Ref: Picture1]

image

[Picture1]

Enabling TTL on Cosmos DB Collection:

You can enable TTL on a Cosmos DB collection simply by using Azure Portal –> Cosmos DB collection setting for existing or during creation of  a new collection)

TTL value needs to be set in seconds – if you need 90 days => 60 sec * 60 min * 24 hour * 90 days = 7776000 seconds

image

[Picture2]

Below is a one of the reference architecture in which Cosmos DB – TTL would be essentially useful and viable to any Iot business case:

image

[Picture3]

Hope that was helpful to get some understanding. For more references visit:  Cosmos DB Documentation

Azure Cosmos DB – 429 Too Many Requests

October 6, 2018 .NET, Azure, CosmosDB, Document DB, Microsoft, Performance, Reliability, Resilliancy, Scalability, Visual Studio 2017, VisualStudio, VS2017 No comments

Recently while I was doing Performance Testing in one of the APIs interacting with Cosmos DB, I encountered a problem as Azure Cosmos DB API’s started returning Http Code 429.  Http Status Code 429 indicates that too many request been received or request rate is very large. This error would happen when we have concurrent users trying to write or read from same cosmos db collection.

Following diagram covers the architecture of the performance test I am performing:

image

Based on analysis it found out to be the Throttling happening from Azure Cosmos DB, as we make requests that may use more than provisioned Request Units(RU) per second. We were using default Cosmos DB configuration for a fixed collection of 1000 RU’s per second which is sufficient enough for a 500 reads and 100 writes for a 1 kb file. You can refer more about Request Units from Azure Docs.

image

 

 

 

Solution(s):

1. Now first logical step we can do is to get rid off this error by increasing the Throughput for the collection.  I am going to increase to 10000 RU/s maximum allocatable for a Storage Capacity: Fixed.   This should ideally improve the Throughput for 250 or more virtual users hitting.

image

2. Second logical step is to improve the code: Improve the connection parameters in the Document DB SDK –> DocumentDbClient. For this I referred to the Microsoft Docs: Performance tips for Azure Cosmos DB and .NET

Providing optimum values to the following Properties in RetryOption class   to be passed as parameter to Connection Policy.

image

 

In my case I provided a value of 30 to give ultimate results:

new RetryOptions() { MaxRetryAttemptsOnThrottledRequests = 30, MaxRetryWaitTimeInSeconds = 30  }

That should resolve most of the 429 issues when dealing with Cosmos DB SDK

Introduction to NDepend : Static Code Analysis Tool

June 16, 2018 .NET, .NET Core, .NET Framework, ASP.NET, Best Practices, C#.NET, Code Analysis, Code Quality, Dynamic Analysis, Emerging Technologies, Help Articles, Microsoft, Static Analysis, Tech-Trends, Tools, Tools, Visual Studio 2017, VisualStudio, Windows No comments , , , , , ,

As a developer, you always have to take the pain of getting adapted to the best practices and coding guidelines to be followed as per the organizational or industrial standards.  Easy way to ensure your coding style follows certain standard is to manually analyze your code or use a static code analyzer like FxCop, StyleCop etc. Earlier days I have been a fan of FxCop as it was free and it provides me all necessary general guidelines in terms  of improving my solution.

In this modern world of programming everything needs to be automated, as it saves time and money in terms of automating repetitive tasks and improves efficiency. This is where static code analysers coming effective.

What is Static Code Analysis?

Static program analysis is the analysis of computer software that is performed without actually executing programs, on some version of the program source code, and in the other cases, some form of the object code or intermediate compiled code .

Sophistication of static program analysis increases is based on how deep they analyze in terms of behavior of individual statements and declarations, to analyzing the entire source code.

PS: Analysis performed on executing programs is known as dynamic analysis.

In this article I will give you an overview of one such premier static code analysis tool that can be used for your daily development routine plus use it for CI integration for DevOps efficiency.

NDepend:

NDepend is a static analysis tool for .NET, specifically for managed code:  NDepdend supports a large number of code metrics, allowing to visualize dependencies using directed graphs and dependency matrix. It also performs code base snapshots comparisons, and validation of architectural and quality rules.

The important capabilities of NDepend are:

  • Dependency Visualization through dependency matrix and graphs.
  • Analyse and generate software quality metrics – as per the documentation it supports 82 quality metrices.
  • Declarative rule support through LINQ queries, and it is called CQLinq and comes with a large number of predefined CQLinq rules.
  • Integration support for Cruise Control.Net, SonarCube, am City. Code rules can be configured to be checked automatically in Visual Studio or during continuous integration(CI).

License: NDepend is a commercial tool with licensing options as below:

  1. Developer seats – $477 approx. / per seat.
  2. Build Machine seats  – $955 approx. / per seat.

** You could get volume discount if you bulk procure your licenses.

Installation: 

Once you obtained license you will able to download NDepend_2018.1.1.9041.zip, is latest version available while I write this article. Extract the zip file into your local folder, you could see the different packages/executables within the package.

image

1.) NDepend.Console    – Command line program to execute NDepend analysis.  You would be mostly using this component on CI Build server Help

2.) NDepend.PowerTools –  Helps write your own static analyzer based on NDepend.API, or tweak existing open-source Power Tools. Help

image

3.) NDepend.VisualStudioExtension.Installer – To install NDepend extension as part of Visual studio

image

4.) VisualNDepend – Independent visual environment for managing your NDepend tasks.

image

Visual Tool gives you different options to choose from:

  • You can analyse a Visual Studio Solution or project.
  • Analyse .NET assemblies in a folder.

image

image

image

For the demo purpose our analysis target would be one of the starter project from github –  ContosoUniversity by @alimon808.

image

image

Demo: Summary Report

image

Demo: Application Metrics

image

Demo: Dependency Dashboard:

image

Demo: Interactive Graph

image

Demo: Code Matrix View

image

Demo: Quality Gates Summary

image

Demo: Rules Summary

image

Conclusion:

NDepend is one of the best enterprise grade commercial static analyser seen so far.  There are Visual Studio Code Analysis, FxCop and Stylecop Analyzer tools available but they do not provide extensive level of analysis reports NDepend provides. Being a commercial tool it gives value for money for customers by what they need.  In terms of a day to day developer  or devops lifecycle, you can integrate NDepend in your build process, which could be simple as executing the NDepend Console and reviewing the output. With NDepend’s API it is easy to develop your own custom analysis tools based on CQLinq and NDepend.PowerTools(which is open source). You could find all the detailed help in NDepend documentation.

References:

Getting Started local development with Azure Cosmos DB services – Part 2

May 29, 2017 .NET, .NET Core 1.0, .NET Core 1.0.1, .NET Framework, ASP.NET, Azure, Azure SDK Tools, Azure Tools, Cloud Computing, CodeSnippets, CosmosDB, Document DB, Microsoft, PaaS, SaaS, Visual Studio 2015, Visual Studio 2015 Update 3, Visual Studio 2017, VisualStudio, VS2015, VS2017, Windows, Windows 10, Windows Azure Development, Windowz Azure No comments

In my previous article we discussed about setting local development environment using Cosmos DB Emulator for Windows. With this part 2 of the article, we will cover developing, debugging and integration related aspects of using Cosmos DB Emulator.

Developing with Cosmos DB Emulator

Once you have Cosmos DB emulator installed and running on your machine, you can use any supported Cosmos DB SDK or Cosmos DB REST API to interact with emulator. This process is same as you are using a Cosmos DB cloud service.

Cosmos DB Emulator also provides a build-in visual explorer through which you can view,create and edit collections and documents.

image

Before you integrate Cosmos DB SDK or Cosmos DB REST API you would need to generate master key for authentication. Unlike cloud service, Cosmos DB emulator only support single fixed account and master key.  You would not be able to communicate with Emulator without this master key.

Default Master Key:

Account name: localhost:<port>

Account key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==

PS: This key is only to be used in Emulator. You cannot use the same key for Production(Cosmos DB Cloud Service).

Furthermore, if you want to set your own key. You can go to command line references and run DocumentDB.Emulator.exe with sufficient command switch to set your own key. Remember it should meet the key security requirements. See command-line tool reference for more information.

The Azure Cosmos DB Emulator is installed by default to the C:\Program Files\Azure Cosmos DB Emulator  or C:\Program Files\DocumentDB Emulator  directory.

Once you have account name and key, you are good to go with development and debugging using Azure Cosmos DB emulator.

Let us start looking at how to use CosmosDB SDK. Once you add Cosmos DB SDK for .NET from NUGET sources. You would need to import the following namespaces to reference necessary classes.

 using Microsoft.Azure.Documents;
   
 using Microsoft.Azure.Documents.Client;
   
 using Microsoft.Azure.Documents.Linq;

Simple code to establish connection:

// Connect to the Azure Cosmos DB Emulator running locally use DocumentClient class in : 
DocumentClient client = new DocumentClient(
    new Uri("https://localhost:8081"), 
    "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");

In the above code block we are directly embedding endpoint, key in the source code.But as a suggested approch keeping in mind to easily point to production service would be maintain the key in Web.config appSettings.

   <add value="https://localhost:8081/" key="endpoint"/>
    <add value="C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" key="authKey"/>
 

Add NuGet reference to Microsoft.Azure.DocumentDB  (always use the latest version of the library)

image

For the ease of this article, I am going to use the existing ToDoList sample from DocumentDB Samples provided by Microsoft. You can originally find the same source from C:\Program Files\DocumentDB Emulator\Packages\DataExplorer\quickstart.

image

Copy and Unzip DocumentDB-Quickstart-DotNet.zip and open todo.sln in Visual Studio 2017 and your solution structure will look like below:

image

Now run the application in your Visual Studio.

1. You will see an initial screen:

image

2. Click on Create New:

image

3. New record will be added to your Azure Cosmos DB Emulator:

image

4. To verify in Cosmos DB emulator now open Cosmos DB explorer, click on Collections and Select ToDoList

image

5.Expand Documents and select item with id:da305da3-c1dc-4e34-94d9-fd7f82d26c58

image

Hope this article was helpful for you with initial development.  Share your feedback through comments and share this to your friends and colleagues.

Useful Links:

Getting Started local development with Azure Cosmos DB services – Part 1

May 20, 2017 .NET, Azure, Azure SDK, Azure SDK Tools, Azure Tools, Cloud Computing, Computing, CosmosDB, Data Services, Document DB, Emerging Technologies, KnowledgeBase, Microsoft, PaaS, Visual Studio 2013, Visual Studio 2015, Visual Studio 2017, VS2013, VS2015, VS2017, Windows 10, Windows Azure Development, Windows Server 2012 R2, Windows Server 2016, Windowz Azure 1 comment , ,

Azure Cosmos DB is a multi-API, multi-model highly scalable NoSQL database services from Microsoft Azure platform. In order to develop an application consuming Azure Cosmos DB requires an azure live subscription or emulator in your local machine.

The Azure Cosmos DB Emulator provides a local development/test environment for Azure Cosmos DB development purposes. Using Azure Cosmos DB Emulator, you can develop and test your application locally, without needing an azure subscription or without subscription costs.

With this article I am going to take you through necessary steps and requirements to set up your local environment.

1. Pre-Requisites:

Azure Cosmos DB emulator has the following software and hardware requirements:

  • Software requirements
    • Windows Server 2012 R2, Windows Server 2016, or Windows 10
  • Minimum Hardware requirements
    • 2 GB RAM
    • 10 GB available hard disk space

2. Installation:

  • Download Azure Cosmos DB Emulator   (DocumentDb.Install.msi)   ** do not get confused by the name. Azure Cosmos DB is a super set of Document DB, and the DocumentDb emulator they tweaked a bit to support Cosmos Db.
  • Install DocumentDb.Install.msi

Additionally Azure CosmosDB emulator can be run on Docker for Windows. After installing Docker for Windows, you can pull the Emulator image from Docker Hub.

docker pull microsoft/azure-documentdb-emulator

imageimageimage

3. Start/Launch Azure Cosmos DB Emulator:

image image

After some time you can see the emulator started. When the Azure Cosmos DB emulator launches it will automatically open the Azure Cosmos DB Data Explorer in your browser.

The address will appear as https://localhost:8081/_explorer/index.html

Incase you have closed browser and later would like to open the explorer again, you can open the Data Explorer by right clicking on the taskbar menu.

image

image

Now you can write some sample app to try it, or download already created sample applications from Microsoft depending on the preferred platform of your choice.

4. Limitation of Azure Cosmos DB Emulator: (or Differences between Azure Cosmos DB Emulator vs Real Cosmos DB Cloud Service)

Since the Azure Cosmos DB Emulator provides an emulated environment running on a local developer workstation, there are some fundamental differences between the emulator and an Azure Cosmos DB account in the cloud:

The following table is also helpful in determining when to use Cosmos DB Emulator and when direct cloud service. Depending on the choice of requirement, you would need to use associated services efficiently.


Cosmos DB Emulator Cosmos DB Cloud Service
Supports only a single fixed account and a well-known master key. Key regeneration is not possible. Supports multiple accounts and different master keys. You can regenerate keys any time from Azure Portal.
Non scalable Highly scalable
Does not support larger data sets Support for large data sets
Does not simulate consistency levels Different Consistency levels available
Does not simulate multi-region replication Configurable as part of the platform, as needed basis.
Does not support quota override feature Supports document size limit increases, increased partitioned collection storage etc.
Might not support most recent changes to Cosmos Db platform Most recent platform update will be available.

Hope this article was helpful for your initial start. If you would need to understand further on Azure Cosmos DB development follow the links. I will be writing further insights in later sessions.

.NET Framework 4.7–Released for All versions of Windows

May 3, 2017 .NET, .NET 4.7, .NET Framework, .NET Framework 4.7, ASP.NET, ASP.NET MVC, C#.NET, Caching, Cryptography, Extensions, Microsoft, Performance, Security, Visual Studio 2013, Visual Studio 2015, Visual Studio 2017, VisualStudio, VS2012, VS2013, VS2015, WCF, Web API, Web API v2.0, Windows, Windows 10, Windows 7, Windows 8, Windows 8.1, WinForms, WPF No comments

Microsoft has released next version of .NET Framework (do not get confused with .NET Core) .  Though the .NET Framework 4.7 was released as part of Windows 10 Creators Update a month ago. You can now install the .NET Framework 4.7 on other versions of Windows

Download the: .NET Framework 4.7  – Web installer  |  Offline Installer

.NET Framework 4.7 Developer Pack  In order to add support for .NET Framework 4.7 in Visual Studio 2012 or later we need to install Developer Pack.

Windows Versions and Support:

The .NET Framework 4.7 is supported on the following Windows versions:

  • Windows 10 Creators Update (included in-box)
  • Windows 10 Anniversary Update
  • Windows 8.1
  • Windows 7 SP1

The .NET Framework 4.7 is supported on the following Windows Server versions:

  • Windows Server 2016
  • Windows Server 2012 R2
  • Windows Server 2012
  • Windows Server 2008 R2 SP1

New Features in .NET Framework 4.7:

On a high-level below are the set of new features introduced in following areas:

Core

Networking  Default operating system support for TLS protocols*

ASP.NET

  • Object Cache Extensibility  (plug in new implementations of an object cache for an ASP.NET application by using the new ICacheStoreProvider interface. )
  • Memory monitoring (Developers can now write their own memory monitors to replace the default by using the ApplicationMonitors.MemoryMonitor property.)
  • Memory Limit Reactions. (Developers can now replace or supplement the default behavior by subscribing IObserver implementations to the application’s memory monitor.

Windows Communication Foundation (WCF) 

  • Ability to configure the default message security settings to TLS 1.1 or TLS 1.2
  • Improved reliability of WCF applications and WCF serialization

Windows FormsHigh DPI support

Windows Presentation Foundation (WPF)

  • Support for a touch/stylus stack based on Windows WM_POINTER messages
  • New implementation for WPF printing APIs

Also improvements in :

  • High DPI support for Windows Forms applications on Windows 10
  • Touch support for WPF applications on Windows 10
  • Enhanced cryptography support
  • Support for C# 7 and VB 15, including ValueTuple
  • Support for .NET Standard 1.6
  • Performance and reliability improvements

 

Additional References: