Azure Monitor: Management Tips from the Field

Since I’ve started working in Azure full time for the last few years I’ve now seen a number of environments. This post will outline my Azure Monitor Management Tips to make your life easier. And how you should set it up depending on your needs. Given my focus has largely been on Monitoring, these tips are geared towards Log Analytics workspaces, linking Azure Automation accounts and Alerting.

Subscriptions and Log Analytics Workspaces

Absolutely you should be using multiple subscriptions to segregate applications, cost centers or any other number of reasons. What you should not do is setup a Log Analytics workspace per subscription except for very specific reasons.

Workspaces: Current Recommendations

My current recommendation for setting up workspaces is to do one for Production, Dev, UAT, etc based on your environments needs. For instance you may want to alert or be notified if something is wrong in DEV but you don’t necessarily want to collect the data at the interval you would for Production. Take IaaS VMs. In Dev you may only want a collection interval of every 5 minutes, but in Production you want it to be 60 seconds or higher. Having two different workspaces allows you to set those settings specifically for each workspace.

For now your workspace should be setup in East US, as at present all Log Analytics solutions are available in that region. Though workspaces have expanded to new regions not all solutions have expanded with them.

Workspaces: Don’t

As I mentioned above you should not install a workspace per subscription unless you have very specific reasons to. Like completely segregated Management/Monitoring teams per infra or application. In this scenario though the client had 18 subscriptions, and 18 workspaces and 1 Azure Management team managing it all. What happened next was they wanted consolidation of everything. Thats where my Azure Automation Update Management Workbook came from. Prior to workbooks however, the only way to aggregate multiple Log Analytics workspaces was using an Azure Dashboard and joins or unions. They wanted a dashboard that monitored a bunch of services like Splunk, Carbon Black etc from VMs across all 18 subscriptions. This ended up being a huge pain to create. You can see the sample code below.

let workspace1 = workspace("workspace1").ConfigurationData
| where SvcDisplayName == "SplunkForwarder Service"
| distinct Computer, SvcDisplayName, SvcState
| join kind= rightouter( workspace("workspace1").Heartbeat | distinct Computer, OSType, SubscriptionId) on Computer
| project Computer=Computer1, SvcDisplayName, SvcState, OSType, SubscriptionId
| join kind= leftouter ( workspace("workspace1").Perf | where ObjectName == "Process" and InstanceName == "splunkd" | distinct Computer, InstanceName) on Computer;
//
let workspace2 = workspace("workspace2").ConfigurationData
| where SvcDisplayName == "SplunkForwarder Service"
| distinct Computer, SvcDisplayName, SvcState
| join kind= rightouter( workspace("workspace2").Heartbeat | distinct Computer, OSType, SubscriptionId) on Computer
| project Computer=Computer1, SvcDisplayName, SvcState, OSType, SubscriptionId
| join kind= leftouter ( workspace("workspace2").Perf | where ObjectName == "Process" and InstanceName == "splunkd" | distinct Computer, InstanceName) on Computer;
//
worksapce1 | union workspace2, workspace3, workspace4

This is just 2 queries for 2 of the workspaces and its just one service. They wanted 5+ services across 18 workspaces. You can now imagine how difficult it became to make a change to the queries across all 18 queries. Oh and they also wanted to compare the service count with the count of machines from the heartbeat table. So I then had to go back and add all the joins to each query. Granted this is now easier to accomplish with workbooks and the ability to dynamically select resources.

Azure Automation and Log Analytics

To link an Azure Automation account and Log Analytics workspace they must be in the same region. They do not have to be in the same subscription though. I definitely recommend linking an Azure Automation account with Log Analytics in IaaS scenarios. As you get Update Management, Change Tracking and Inventory.

Azure Automation: Don’t

They also had 1 Azure Automation account linked to each of these workspaces. They had automated scripts that changed settings in Azure Backup vaults. It was discovered that this script was not functioning correctly. So someone had to then go manually check every Azure Backup vault, and every script in 18 Azure Automation Accounts. Instead of having one production Azure Automation account handling this script.

Alerting

As a consequence of having 18 workspaces, when it came time to setup alerts, alerts had to be created for each workspace. Because you need a workspace to alert on RAM usage, its best to use a workspace to create IaaS VM Metric alerts. This wouldn’t be a problem. Except now we had to create 5 plus metric alerts for every workspace, instead of just one workspace. The metrics, thresholds, and performance collection was the same on every workspace, so its not like we needed different alerts for different workspaces. Also because they wanted to use the ITSM connector, the connector had to be setup in one subscription and an Action Group that uses that connector had to be setup in each subscription for the alerts to point at.

Setting up your environment this way may sound like a good idea, but as time goes on more and more issues come up that become more and more time consuming to resolve than properly setting up your environment the first time.

Summary: Azure Monitor Management Tips

Setup one workspace for your production environment, and if you need different settings for Dev/UAT/QA setup workspaces for those environments as well. The only reason other than non prod workspaces to have more than one workspace, is if you have multiple Ops teams for different environments and they never need to access the same information, action groups and alerts.

Setup one Automation Account with your production workspace.

Setup action groups and alerts against your prod workspace.