Welcome to the World of Tomorrow:

We are in an emerging world of technologies and innovation, where IoT cannot stand alone.

The Internet-of-Things(a.k.a IoT) provides us with lots of sensor data/information. However, this sensor information by themselves does not add value unless we can turn them into actionable, contextualized information useful for mankind. Big data, data visualization and Machine Learning(ML) techniques allow us to gain new insights by learning, batch-processing and information analysis(online/off-line). Real-time sensor data analysis and decision-making are often done manually but to make it scalable, it is preferably automated.

Artificial Intelligence(AI) provides us the set of framework and tools to go beyond typical real-time decision and automation use cases for IoT. Who knows we may end up in a world where humans have less work to do like in Sci-Fi Hollywood movies. Whatever happens will be the best towards for our next generations and we have the responsiblity to build a better world for them.

The intent of this blog is to provide you latest insight into what is happening in the Cloud, IoT & AI industry. Keep reading, keep contributing and please share as well.

Introduction to Data Science

We all have been hearing the term Data Science and Data Scientist occupation become more popular these days. I thought of sharing some light into this specific area of science, that may seem interesting for rightly skilled readers of my blog. 

Data Science is one of the hottest topics on the Computer and Internet  nowadays. People/Corporations have gathered data from applications and systems/devices until today and now is the time to analyze them. The world wide adoption of Internet of Things has also added more scope analyzing and operating on the huge data being accumulated from these devices near real-time.

As per the standard Wikipedia definition goes Data science, also known as data-driven science, is an interdisciplinary field about scientific methods, processes and systems to extract knowledge or insights from data in various forms, either structured or unstructured, similar to data mining.”.

Data Science requires the following skillset:

  • Hacking Skills
  • Mathematics and Statistical Knowledge
  • Substantive Scientific Expertise

aoz1BJy

[Image Source: From this article by Berkeley Science Review.]

Data Science Process:

Data Science process involves collecting row data, processing data, cleaning data, data analysis using models/algorithms and visualizes them for presentational approaches.  This process is explained through a visual diagram from Wikipedia.

Data_visualization_process_v1

[Data science process flowchart, source wikipedia]

Who are Data Scientist?

Data scientists use their data and analytical ability to find and interpret rich data sources; manage large amounts of data despite hardware, software, and bandwidth constraints; merge data sources; ensure consistency of datasets; create visualizations to aid in understanding data; build mathematical models using the data; and present and communicate the data insights/findings.

They are often expected to produce answers in days rather than months, work by exploratory analysis and rapid iteration, and to produce and present results with dashboards (displays of current values) rather than papers/reports, as statisticians normally do.

Importance of Data Science and Data Scientist:

“This hot new field promises to revolutionize industries from business to government, health care to academia.”

The New York Times

Data Scientist is the sexiest job in the 21st century as per Harward Business Review.

McKinsey & Company projecting a global excess demand of 1.5 million new data scientists.

What are the skills required for a Data Scientist, let me share you a visualization through a Brain dump.

FxsL3b8

I thought of sharing an image to take you through the essential skill requirements for a Modern Data Scientist.

So what are you waiting for?, if you are rightly skilled get yourselves an Data Science Course.

Informational  Sources:

Microsoft Build Tour – Hyderabad, 21,22 June 2017

The Microsoft Build Tour brings the best of Microsoft Build right to you. Developers building Line of Business or consumer applications using Cloud or traditional desktop technologies will hear firsthand from Microsoft experts about the latest technical news and updates from the Microsoft Build conference.

Date & Timings:
· DAY1 – Wednesday, June 21, 2017 9:00 AM – 5:30 PM
· DAY2 – Thursday, June 22, 2017 9:00 AM – 5:30 PM (Developer Hackathon)

Venue:
Trident Hyderabad
Survey No.64,
Hitech City Main Road,
HITECH City,
Near Cyber Towers,
Madhapur, Hyderabad,
Telangana 500081

AGENDA DAY 1
08:30 – 09:00Registration
09:00 – 09:50Opening keynote shares the future of Microsoft’s Cloud, AI, and Windows technologies.
10:00 – 12:30Morning sessions dive deep into the latest Cloud tech including serverless, containers, AI, bots, Web, .NET, and cross-platform development.
12:30 – 14:00Lunch Break
14:00 – 14:25Technical discussion from your local Microsoft Experts. More details to come!
14:30 – 16:00Afternoon sessions bring in-depth discussions and demos of Windows 10 features, UWP, Windows Store, and the latest devices.
16:00 – 17:30Q&A Session with Microsoft Experts and Developers
AGENDA DAY 2
08:30 – 09:00Registration / Team Assignments / Breakfast
09:00 – 09:30Introduction
09:30 – 16:00Hacking
16:00 – 17:00Presentations & Closing

REGISTRATION LINKS: DAY1 SESSIONS | DAY2 DEVELOPER HACKATHON

Getting Started local development with Azure Cosmos DB services – Part 2

In my previous article we discussed about setting local development environment using Cosmos DB Emulator for Windows. With this part 2 of the article, we will cover developing, debugging and integration related aspects of using Cosmos DB Emulator.

Developing with Cosmos DB Emulator

Once you have Cosmos DB emulator installed and running on your machine, you can use any supported Cosmos DB SDK or Cosmos DB REST API to interact with emulator. This process is same as you are using a Cosmos DB cloud service.

Cosmos DB Emulator also provides a build-in visual explorer through which you can view,create and edit collections and documents.

image

Before you integrate Cosmos DB SDK or Cosmos DB REST API you would need to generate master key for authentication. Unlike cloud service, Cosmos DB emulator only support single fixed account and master key.  You would not be able to communicate with Emulator without this master key.

Default Master Key:

Account name: localhost:<port>

Account key: C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==

PS: This key is only to be used in Emulator. You cannot use the same key for Production(Cosmos DB Cloud Service).

Furthermore, if you want to set your own key. You can go to command line references and run DocumentDB.Emulator.exe with sufficient command switch to set your own key. Remember it should meet the key security requirements. See command-line tool reference for more information.

The Azure Cosmos DB Emulator is installed by default to the C:\Program Files\Azure Cosmos DB Emulator  or C:\Program Files\DocumentDB Emulator  directory.

Once you have account name and key, you are good to go with development and debugging using Azure Cosmos DB emulator.

Let us start looking at how to use CosmosDB SDK. Once you add Cosmos DB SDK for .NET from NUGET sources. You would need to import the following namespaces to reference necessary classes.

 using Microsoft.Azure.Documents;
   
 using Microsoft.Azure.Documents.Client;
   
 using Microsoft.Azure.Documents.Linq;

Simple code to establish connection:

// Connect to the Azure Cosmos DB Emulator running locally use DocumentClient class in : 
DocumentClient client = new DocumentClient(
    new Uri("https://localhost:8081"), 
    "C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==");

In the above code block we are directly embedding endpoint, key in the source code.But as a suggested approch keeping in mind to easily point to production service would be maintain the key in Web.config appSettings.

   <add value="https://localhost:8081/" key="endpoint"/>
    <add value="C2y6yDjf5/R+ob0N8A7Cgv30VRDJIWEHLM+4QDU5DE2nQ9nDuVTqobD4b8mGGyPMbIZnqyMsEcaGQy67XIw/Jw==" key="authKey"/>
 

Add NuGet reference to Microsoft.Azure.DocumentDB  (always use the latest version of the library)

image

For the ease of this article, I am going to use the existing ToDoList sample from DocumentDB Samples provided by Microsoft. You can originally find the same source from C:\Program Files\DocumentDB Emulator\Packages\DataExplorer\quickstart.

image

Copy and Unzip DocumentDB-Quickstart-DotNet.zip and open todo.sln in Visual Studio 2017 and your solution structure will look like below:

image

Now run the application in your Visual Studio.

1. You will see an initial screen:

image

2. Click on Create New:

image

3. New record will be added to your Azure Cosmos DB Emulator:

image

4. To verify in Cosmos DB emulator now open Cosmos DB explorer, click on Collections and Select ToDoList

image

5.Expand Documents and select item with id:da305da3-c1dc-4e34-94d9-fd7f82d26c58

image

Hope this article was helpful for you with initial development.  Share your feedback through comments and share this to your friends and colleagues.

Useful Links:

Getting Started local development with Azure Cosmos DB services – Part 1

Azure Cosmos DB is a multi-API, multi-model highly scalable NoSQL database services from Microsoft Azure platform. In order to develop an application consuming Azure Cosmos DB requires an azure live subscription or emulator in your local machine.

The Azure Cosmos DB Emulator provides a local development/test environment for Azure Cosmos DB development purposes. Using Azure Cosmos DB Emulator, you can develop and test your application locally, without needing an azure subscription or without subscription costs.

With this article I am going to take you through necessary steps and requirements to set up your local environment.

1. Pre-Requisites:

Azure Cosmos DB emulator has the following software and hardware requirements:

  • Software requirements
    • Windows Server 2012 R2, Windows Server 2016, or Windows 10
  • Minimum Hardware requirements
    • 2 GB RAM
    • 10 GB available hard disk space

2. Installation:

  • Download Azure Cosmos DB Emulator   (DocumentDb.Install.msi)   ** do not get confused by the name. Azure Cosmos DB is a super set of Document DB, and the DocumentDb emulator they tweaked a bit to support Cosmos Db.
  • Install DocumentDb.Install.msi

Additionally Azure CosmosDB emulator can be run on Docker for Windows. After installing Docker for Windows, you can pull the Emulator image from Docker Hub.

docker pull microsoft/azure-documentdb-emulator

imageimageimage

3. Start/Launch Azure Cosmos DB Emulator:

image image

After some time you can see the emulator started. When the Azure Cosmos DB emulator launches it will automatically open the Azure Cosmos DB Data Explorer in your browser.

The address will appear as https://localhost:8081/_explorer/index.html

Incase you have closed browser and later would like to open the explorer again, you can open the Data Explorer by right clicking on the taskbar menu.

image

image

Now you can write some sample app to try it, or download already created sample applications from Microsoft depending on the preferred platform of your choice.

4. Limitation of Azure Cosmos DB Emulator: (or Differences between Azure Cosmos DB Emulator vs Real Cosmos DB Cloud Service)

Since the Azure Cosmos DB Emulator provides an emulated environment running on a local developer workstation, there are some fundamental differences between the emulator and an Azure Cosmos DB account in the cloud:

The following table is also helpful in determining when to use Cosmos DB Emulator and when direct cloud service. Depending on the choice of requirement, you would need to use associated services efficiently.


Cosmos DB EmulatorCosmos DB Cloud Service
Supports only a single fixed account and a well-known master key. Key regeneration is not possible.Supports multiple accounts and different master keys. You can regenerate keys any time from Azure Portal.
Non scalableHighly scalable
Does not support larger data setsSupport for large data sets
Does not simulate consistency levelsDifferent Consistency levels available
Does not simulate multi-region replicationConfigurable as part of the platform, as needed basis.
Does not support quota override featureSupports document size limit increases, increased partitioned collection storage etc.
Might not support most recent changes to Cosmos Db platformMost recent platform update will be available.

Hope this article was helpful for your initial start. If you would need to understand further on Azure Cosmos DB development follow the links. I will be writing further insights in later sessions.

Azure in Germany–a complete EU cloud computing solution

With my earlier article Azure in China, it came in to my interest to look for any other country/region specific independent cloud data center requirements.  I came across Azure for US Govt(Similar to Amazon Govt Cloud) instance and Azure Germany data center.  For this article context I will be covering only Azure in Germany.

What is Azure Germany?

Just like regional regulatory requirements in China, Germany also wanted a completely locally owned/managed Azure Data Center for EU/EFTA/UK requirements. This is also to ensure stricter access control and data access policy measurements. This  approach is to enable organizations doing business in EU/EFTA and UK can better harness the power of cloud computing.

  • All customer data and related applications and hardware reside in Germany
  • Geo-replication between datacenters in Germany to support  business continuity
  • Highly secured datacenters provide 24×7 monitoring
  • It meets all Public sector or restricted industry requirements
  • Follows all Compliance requirements for EU/EFTA and UK.
  • Lower cost, locally accessible  within your business locations in Germany/EU.

“ Azure Germany is an isolated Azure instance in Germany, independent from other public clouds.”

Who controls it?

An independent data trustee controls access to all customer data in the Azure Germany datacenters. T-Systems International GmbH, a subsidiary of Deutsche Telekom and an experienced, well-respected IT provider incorporated in Germany, serves as trustee, protecting disclosure of data to third parties except as the customer directs or as required by German law.

** Even Microsoft does not have access to customer data or the datacenters without approval from and supervision by the German data trustee.

What Compliance?

Azure Germany has an ongoing commitment to maintaining the strictest data protection measures, so organizations can store and manage customer data in compliance with applicable German laws and regulations, as well as key international standards. Additional compliance standards and controls that address the unique role of the German data trustee will be audited over time. Refer to: Microsoft Trust Center compliance.

[Source : Microsoft Azure]

Useful Links:

Azure in China

Microsoft Azure presence in China is always a question when there is a need for any customer to deploy azure applications specifically for Chinese Regional customers.

Recently I had an interaction with a Microsoft Certified Trainer, who carelessly said Azure only uses Chinese partner based environment is only for serving CDN (Content Delivery Network) needs, partial knowledge is difficult to convince. Being a Certified Trainer, he should have known these. As I recollect my earlier experience with azure, I was aware that Microsoft has partnered with 21Vianet , to setup a Chinese instance of Azure with most possible IaaS, PaaS and SaaS offerings are available to Chinese customers. At the same time data would reside within Chinese governments data privacy and storage regulations.

Similarly other competitor cloud providers such as IBM also partnered with 21Vianet to setup their BlueMix instances in China.

Microsoft Azure in China

General availability of Azure in China through 21vianet partnership has been announced in March 26th 2014. These Azure Services in Chinese region will be operated and sold by 21vianet.

Quoting to Doug Hauger GM- National Cloud Programs, Microsoft

Windows Azure, operated by 21Vianet, is based on the same technology as Microsoft’s global Windows Azure service and offers a reliable, flexible and value-based service to bring Microsoft’s latest cloud technologies and benefits to Chinese customers and propel forward technological development in China as a whole.

Why Partnership?

As per Chinese Government(PRC) Cyber Security Law and Governance policies there is a clear restriction for a foreign entrant to obtain license and provide services to Chinese customers.  All the data relating to Chinese customers should reside within Chinese Data Centres, and foreign companies would have to pass on the operational ownership to a Chinese company in order to operate within PRC(Peoples Republic of China).

In Additional to that on November’2016 new amendments attached to new PRC Cyber Security Law (effective 1st June 2017).

As per DLA PIPER

The new law will come into force on 1 June 2017 and has significant implications for the data privacy and cybersecurity practices of both Chinese companies and international organisations doing business in China.

The new PRC Cybersecurity Law intends to combat online fraud and protect China against Internet security risks. In short, it imposes new security and data protection obligations on “network operators”; puts restrictions on transfers of data outside China by “key information infrastructure operators”; and introduces new restrictions on critical network and cybersecurity products.  [Read more from Source]

Where is Azure in China?

You can access Azure in China instance through a separate portal https://www.azure.cn. Azure Portal is available at https://portal.azure.cn/. Pricing information is available on http://www.azure.cn/pricing/pia/  (click on English to get an English version).

Below are two data centre regions Azure is available in China.

image

Services Offered in Azure China Instances:

Not all all services of Azure including Mobile Services, Analytics and IoT Suite are available for Chinese regions. I will try to cover a specific list of services that are available for China region.Azure China is a few versions behind regular Azure and not all services are supported. Some of the services supported are:

  • Compute – Virtual Machines (For IaaS Services)
  • Compute – Cloud Service (For PaaS Services)
  • Compute – Web Site
  • Data Services – Storage (For Blobs, tables and Queues)
  • Data Services – SQL Database
  • Data Service – HDInsight
  • SQL Database – SQL Import/ExportNetworks – Virtual Network (To communicate between cloud services via Local networks)
  • Service Bus
  • Active Directory
  • Access Control Service – which has its own UI and namespace
  • Caching – only dedicated cache is supported
  • Mobile Service
  • Media Service
  • CDN
  • Traffic Manager
  • Azure IoT Suite
  • Notification Center
  • Event Center
  • Service Fabric
  • Batch Service
  • Flow Analytics, Power BI
  • Document DB, SQL Database, SQL Data Warehouse, MySQL
  • and many more

image

A list of supported and not supported services can be found at (This is in Chinese, and it is a working document, use google translate to take you through all steps necessary for you to understand services offered in Azure China) http://www.windowsazure.cn/documentation/articles/developerdifferences/#readyonwacn

This also includes all the endpoints that are required for developers to integrate/develop applications targeting China region.

Hope this information was helpful to you. I would assume it would be helpful for some who are specifically looking forward for providing technology solutions for Chinese customers through Azure Cloud.

Sources: