Azure Activity Log Export and Journey of Log Analytics Legacy Log Profile

Azure Activity Log is one of the core log sources to ingest in the Microsoft ecosystem to SIEM (such as Microsoft Sentinel). According to Microsoft: “It’s a subscription log that provides insights into subscription-level events that occur in Azure, including events from Azure Resource Manager operational data, service health events, write operations taken on the resources in subscription, and the status of activities performed in Azure”.

Background

In the early days of Microsoft Sentinel, the legacy profile was the only way to ingest Azure Activity Log from Azure subscription level to Azure Log Analytics. There are a few scenarios, based on my experience, that might have led to using the legacy log profile in log ingestion:

Azure Activity Logs export configuration has been made on time where there weren’t any other options available
- Log Analytics legacy log profiles haven’t been transitioned to Azure diagnostic settings
The existing Log Analytics workspace has been used when the Sentinel instance has been deployed and the legacy log profile has been used to send logs to the Log Analytics workspace
Activity Log export was made directly on Log Analytics UI which used Log Analytics ‘Data Sources’ API

The purpose of this blog post is to show how to find out if a legacy log profile is used in your environment and how to address the issue if it is. I’ve seen this issue recently in a few large (+100 subscription environments).

What’s Legacy Log Profile?

In the early days of Azure Log Analytics, the Azure diagnostic setting was supported by only a fraction of the resources. In those days, you needed to configure the Azure Activity log feed through the legacy log profile. Behind the scenes there was (and still is) Log Analytics ‘Data Sources’ API.

This API still works and you are able to use it to manage log exports on some parts but it doesn’t have as granular controls as Azure diagnostic settings. The latter one (diagnostic settings) is nowadays a preferred way to send Azure Activity log exports to necessary targets (Storage Account, Log Analytics, Event Hub). New experience provides better functionality and consistency with resource logs.

Log Analytics integration can be managed only through Log Analytics Data Source API
- Take into account that configuration is made to Log Analytics, not in the Azure subscription level like diagnostic settings configuration

Storage account & Event Hub integration settings are saved to the legacy log profile but can be configured through PowerShell

Log Analytics Integration

The integration settings are maintained at the Azure Log Analytics end and are available only through the API mentioned above.

Azure management API can be used for configuring Log Analytics integration with the following URI:

https://management.azure.com/subscriptions/<Subscription Id>/resourcegroups/oms-rg/providers/Microsoft.OperationalInsights/workspaces/<LAWS Name>/dataSources/AzureActivityLog?api-version=2020-08-01'

Example commands for the API to create and delete the legacy log profile

## Body
$body = @"
{
  "properties": {
    "LinkedResourceId": "/subscriptions/<Subscription Id>/providers/microsoft.insights/eventtypes/management"
  },
  "kind": "AzureActivityLog"
}
"@

## Set Legacy Log Profile
$SetLegacyProfile = Invoke-restmethod -Method Put -uri 'https://management.azure.com/subscriptions/<Subscription Id>/resourcegroups/<LAWS-rg>/providers/Microsoft.OperationalInsights/workspaces/<LAWS Name>/dataSources/AzureActivityLog?api-version=2020-08-01' -body $body -ContentType "application/json" -Headers $h

## Delete Legacy Log profile
$DeleteLegacyProfile = Invoke-restmethod -Method DELETE 'https://management.azure.com/subscriptions/<Subscription Id>/resourcegroups/<LAWS-rg>/providers/Microsoft.OperationalInsights/workspaces/<LAWS Name>/dataSources/AzureActivityLog?api-version=2020-08-01' -ContentType "application/json" -Headers $h

Storage Account & Event Hub Integration

The legacy log profile for the storage account & event hub can be configured through PowerShell.

If you would like to test this in your environment here is an example of PowerShell commands that can be used for creating the legacy log profile and configuring Azure Activity Log to be sent to the Azure storage account & Event Hub.

# Settings needed for the new log profile
$logProfileName = 'EntLogTest '
$locations = (Get-AzLocation).Location
$locations += "global"
$subscriptionId = "<insert sub Id>"
$resourceGroupName = "EventHub-rg"
$eventHubNamespace = "fetads-eh"

# Build the service bus rule Id from the settings above
$serviceBusRuleId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.EventHub/namespaces/$eventHubNamespace/authorizationrules/RootManageSharedAccessKey"

# Build the Storage Account Id from the settings above
$storageAccountId = "/subscriptions/$subscriptionId/resourceGroups/$resourceGroupName/providers/Microsoft.Storage/storageAccounts/$storageAccountName"

Add-AzLogProfile -Name $logProfileName -Location $locations -StorageAccountId  $storageAccountId -ServiceBusRuleId $serviceBusRuleId

The Problem with Legacy Log Profile

If you have the legacy log profile configured there are a few issues with it:

It doesn’t contain all the same audit log categories as Azure diagnostic settings
Management of the settings is done only through API
It will be most probably deprecated in the near future
New experience provides better functionality and consistency with resource logs
Most important – you will have gaps in your security monitoring data flows and detection rules might not detect adversary actions in some scenarios

If you have data exported through diagnostics settings there are more options available for different audit categories which can be sent to three (3) different targets:

Log Analytics
Storage account
Event Hub

The difference in events is significant if we compare these data sources and the configuration. The legacy log profile contains only the ‘Administrative’ & ‘Policy’ categories and with diagnostic settings, there is a possibility to send other categories as well. The categories are:

Administrative
Security
ServiceHealth
Alert
Recommendation
Policy
Autoscale
ResourceHealth

From the pictures below is visible that even if the subscription is basically empty without resources there is a double amount of events when all categories are selected and configured through diagnostic settings.

union withsource=_TableName AzureActivity
| where _SubscriptionId contains "ed33758f-a00f-4365-8534-938215a3ba9e"
| where TimeGenerated > ago(1d)
| summarize
    Entries = count(),
    Size = sum(_BilledSize),
    last_log = datetime_diff("second", now(), max(TimeGenerated)),
    estimate  = sumif(_BilledSize, _IsBillable == true)
    by _TableName, _IsBillable
| project ['Table Name'] = _TableName, ['Table Size'] = Size, ['Table Entries'] = Entries,
    ['Size per Entry'] = 1.0 * Size / Entries, ['IsBillable'] = _IsBillable
| order by ['Table Size']  desc

More information about the categories is found here. From a security monitoring point of view, all categories are not needed but definitely more than with legacy log profile. I would move forward with Administrative, Security & Policy.

Alerts are included with the Defender for Cloud (MDC) data connector so those are not needed as duplicate events. ServiceHealth, Autoscale & ResourceHealth are more related to operational events than security. The ‘Recommendation’ category is an interesting one but more related to cloud security posture management than potential adversary activities.

The Solution

I’ve seen this issue in multiple environments and it might affect dramatically security monitoring efficiency and detection of possible adversary activities in Azure. What needs to be done is in a nutshell:

Remove legacy log profile configuration
Configure/add diagnostic settings and data export configuration to Azure Monitor Activity Log
- Send audit data where it’s needed (Log Analytics, Storage Account or Event Hub)
Audit & verify that configurations are aligned in all Azure subscriptions

Removing Legacy Log Profile

The legacy log profile is part of the Azure Log Analytics resource and therefore needs to be configured through Log Analytics API.

DELETE 'https://management.azure.com/subscriptions/ed33758f-a00f-4365-8534-938215a3ba9e/resourcegroups/oms-rg/providers/Microsoft.OperationalInsights/workspaces/monae-sentinel/dataSources/AzureActivityLog?api-version=2020-08-01'

After removal, the profile is not found which contains Azure Activity Log audit log export to Log Analytics doesn’t exist anymore

Storage Account & Event Hub Configurations

After removing audit log export to Log Analytics, the other two, storage account & event hub integrations still exist. These are stored underneath different ARM resource provider as you can see below.

 /subscriptions/<Subscription Id>/providers/microsoft.insights/logprofiles/entlogtest

Removal can be done with PowerShell

Remove-AzLogProfile -Name 'entlogtest'

Configure Azure Activity Log Diagnostic Settings

Azure Activity Log diagnostic setting can be configured by Azure Policy which is the easiest way to deploy the needed configuration. The built-in policy contains all audit categories so if you would like to limit or reduce, the audit data some fine-tuning is needed.

Azure Policy reference

If an organization is using IaC to maintain the Azure platform then configuration could be made through IaC and configure audit policy instead to verify that configurations are in place.

Use Azure Policy for Auditing

If IaC is used for configuring Azure Activity Log diagnostic settings you can use Azure Policy to audit that configuration is properly in place. URI for API is below (remember to use version 2021-05-01-preview)

https://management.azure.com/subscriptions/<Subscription Id>/providers/microsoft.insights/diagnosticSettings/?api-version=2021-05-01-preview

Azure Policy contains a built-in policy (Audit diagnostic settings) that is created initially for resource diagnostic settings auditing purposes. Based on my experience, it doesn’t work out of the box with Azure Activity Log because there is metric monitoring included. That being said, I created a custom policy that doesn’t include metrics and the outcome is desired.

From the picture below you can see a custom policy on the first picture from the left, a built-in policy error on the second one, and modification needed on the third one.