Hadoop Configuration Files in HDInsight

 I am keeping these settings for reference purposes and use these settings for quick check.   Mapred-site.xml name value description mapred.system.dir /mapred/system No description mapred.job.tracker jobtrackerhost:9010 No description mapred.jobtracker.taskScheduler org.apache.hadoop.mapred. CapacityTaskScheduler   mapred.job.tracker.http.address jobtrackerhost:50030 No description mapred.local.dir c:\hdfs\mapred\local No description mapred.job.tracker.history.completed.location   No description mapreduce.history.server.embedded true Should job history server be embedded within Job tracker…

4

HDInsight (Hadoop on Azure) Demo: Submit MapReduce job, process result from Pig and filter final results in Hive

In this demo we will submit a WordCount map reduce job to HDInsight cluster and process the results in Pig and then filter the results in Hive by storing structured results into a table. Step 1: Submitting WordCount MapReduce Job to 4 node HDInsight cluster: c:\apps\dist\hadoop-1.1.0-SNAPSHOT\bin\hadoop.cmd jar c:\apps\Jobs\templates\635000448534317551.hadoop-examples.jar wordcount /user/admin/DaVinci.txt /user/admin/outcount The results are stored @ /user/admin/outcount…

0

Powershell script to detect the Windows Azure SDK in Azure Cloud Service VM and fix the Powershell execution issue

When you try to run Powershell commands on Windows Azure Cloud Service (PaaS) VM you may have seen exception as below: PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added. Verify the name of the snap-in and try again. At line:1 char:13 + Add-PSSnapin <<<< Microsoft.WindowsAzure.ServiceRuntime +…

0

Using Fiddler to decipher Windows Azure PowerShell or REST API HTTPS traffic

If you are using publishsettings with Powershell, you may not be able to decrypt HTTPS traffic. Not sure what the problem is with publishsettings based certificates however I decided to create my own certificate using MAKECERT, and use it with Powershell to get HTTPS decryption working in Fiddler. The following steps are described based on…

2

Executing Powershell commands in Windows Azure Web or Worker Role VM Instance return exception

When trying to run Windows Azure Powershell commands in Windows Azure VM (Web Role or Worker Role) instance you will hit the following exception: PS D:\Users\avkash> Add-PSSnapin Microsoft.WindowsAzure.ServiceRuntime Add-PSSnapin : Cannot add Windows PowerShell snap-in Microsoft.WindowsAzure.ServiceRuntime because it is already added. Verify the name of the snap-in and try again. At line:1 char:13 + Add-PSSnapin…

1

Microsoft Server Products and Microsoft License Mobility for Cloud environment

License Mobility Through Software Assurance: With License Mobility through Software Assurance, you can deploy certain server application licenses purchased under your Volume Licensing agreement in an Authorized Mobility Partner’s datacenter. Extend the value of your server application license by deploying them on-premises or in the cloud. Take advantage of the lowest cost computing infrastructure for…

0

How does Windows Azure PowerShell import publishsettings using Import-AzurePublishSettingsFile command?

As you already know you can use Windows Azure Powershell to manage your Windows Azure Service. To get it started,  you need to setup your development machine to be ready to use Windows Azure PowerShell commands and this is done by downloading the publishsetting file from Windows Azure Management Portal about your account first.  …

0