Azure Key Vault is a cloud-based service that allows you to securely store and manage sensitive information such as passwords, certificates, and encryption keys. As with any system that stores sensitive information, it’s important to ensure that access to Azure Key Vault is properly controlled and monitored to prevent unauthorized access by malicious actors.

Lately, I had a case where I needed to dig deep into the Azure Key Vault audit trail for investigation purposes. In this post, I’ll cover how to audit Azure Key Vault for detecting activities from malicious actors with the tools I’ve found the most useful.

Before we dive into the details of Azure Key Vault, I’d like to give a big shout-out to Joosua Santasalo for his invaluable contributions to this topic.

Background

It’s essential to understand how Azure resources are audited and what layers are included. My fellow MVP Joosua Santasalo created the following picture a few years ago and I still find it very useful for describing management & data plane differences in the Azure audit trail.

Management Plane Activities

The Azure management plane audit refers to the auditing of the Azure management plane, which is responsible for managing and controlling Azure resources, services, and operations. The management plane includes Azure Resource Manager, Azure Portal, Azure CLI, Azure PowerShell, and Azure API.

The Azure management plane audit is designed to provide visibility into activities and changes made to the management plane and to help identify potential security risks, compliance violations, and operational issues. It involves reviewing logs and audit trails generated by Azure services and tools that interact with the management plane and analyzing them to identify suspicious or anomalous activity.

The following activities are included in the Azure management plane audit:

1. Authentication and authorization: This includes reviewing activities related to user authentication and authorization, such as changes to Role-based Access Control (RBAC) or changes to resource settings.
2. Resource management: This includes reviewing activities related to the creation, modification, and deletion of Azure resources, such as virtual machines, storage accounts, and databases.
3. Access control: This includes reviewing activities related to access control, such as changes to security groups, network security groups, and firewall rules.
4. Monitoring and alerting: This includes reviewing activities related to monitoring and alerting, such as the configuration of Azure Monitor, alerts, and notifications.
5. Configuration management: This includes reviewing activities related to the configuration of Azure services and tools, such as Azure policies, templates, and resource locks.

Overall, the Azure management plane audit is a critical component of Azure security and helps organizations maintain visibility and control over their Azure environment.

Note: Management plane activities are logged into the ‘AzureActivity’ table in Log Analytics.

Data plane activities

Refer to operations performed on the data stored in Azure services, such as Azure Key Vault or other Azure services. These activities involve accessing, modifying, or deleting data, and they are distinct from management plane activities that involve managing and controlling the Azure environment.

Similar to the Azure management plane audit, the Azure data plane audit is designed to provide visibility into data-related activities and to help identify potential security risks, compliance violations, and operational issues.

The following activities are typically included in the Azure data plane audit:

1. Data access: This includes reviewing activities related to accessing data stored in Azure services, such as reading, writing, and copying data.
2. Data modification: This includes reviewing activities related to modifying data stored in Azure services, such as adding, updating, or deleting data.
3. Data retention and deletion: This includes reviewing activities related to data retention and deletion policies, such as configuring retention periods and deleting data that is no longer needed.
4. Data backup and recovery: This includes reviewing activities related to backing up and restoring data, such as creating backups, scheduling backups, and recovering data from backups.
5. Data encryption and security: This includes reviewing activities related to data encryption and security, such as configuring encryption settings, managing access control, and monitoring security logs.

Overall, the Azure data plane audit is important for maintaining the security, compliance, and operational effectiveness of Azure services that store and process data. By monitoring data-related activities and analyzing audit trails, organizations can identify potential issues and take appropriate actions to mitigate risks, and ensure the integrity and availability of their data.

Note: Data plane activities are logged into the ‘AzureDiagnostics’ table in Log Analytics.

Logs Available in Azure

Besides data & management plane there are other logs available in Azure. The following table lists most of them (source)

Log category	Log type	Usage	Integration
Activity logs	Control-plane events on Azure Resource Manager resources	Process data/security alerts	REST API, Azure Monitor
Azure Resource logs	Frequent data about the operation of Azure Resource Manager resources in subscription	Provides insight into operations that your resource itself performed	Frequent data about the operation of Azure Resource Manager resources in the subscription
Azure Active Directory reporting	Logs and reports	Reports user sign-in activities and system activity information about users and group management	Microsoft Graph
Virtual machines and cloud services	Windows Event Log service and Linux Syslog	Captures system data and logging data on the virtual machines and transfers that data into a storage account of your choice	Windows (using Azure Diagnostics] storage) and Linux in Azure Monitor
Azure Storage Analytics	Storage logging, provides metrics data for a storage account	Provides insight into trace requests, analyzes usage trends, and diagnoses issues with your storage account	REST API or the client library
Network security group (NSG) flow logs	Storage logging provides metrics data for a storage account	Displays information about ingress and egress IP traffic through a Network Security Group	Azure Network Watcher
Application insight	Logs, exceptions, and custom diagnostics	Provides an application performance monitoring (APM) service for web developers on multiple platforms	REST API, Power BI
JSON format shows outbound and inbound flows on a per-rule basis	Microsoft Defender for Cloud alerts, Azure Monitor logs alerts	Provides security information and alerts	REST APIs, JSON

Solutions for Security Monitoring

Microsoft Defender for Cloud (MDC)

MDC provides Cloud Workload Platform Protection (CWPP) layer for multiple services such as Azure Key Vault. I’m not covering the full Defender for Cloud enhanced protection layer here but rather focusing on Azure Key Vault protection instead.

What are the benefits of using MDC for Key Vault?

One of the key ways that Microsoft Defender for Cloud protects Azure Key Vaults is by providing continuous monitoring of Key Vault activities, and by using machine learning to detect suspicious or anomalous behavior. This includes monitoring for unauthorized access attempts, unusual login patterns, and other indicators of potential threats.

Additionally, Microsoft Defender for Cloud provides threat intelligence that helps to identify known threats and vulnerabilities that may affect Key Vaults. This includes leveraging the Microsoft Intelligent Security Graph, which aggregates threat data from multiple sources to provide a comprehensive view of the threat landscape.

Posture Management

MDC provides a wide range of Azure Policies for Key Vault posture management. The full list of available built-in policies is found in Azure Policies for Key Vault.

Threat Protection

When you enabled Defender for Cloud protection for Azure Key Vault you will get protection mainly for external threats. The list of the Azure Keu Vault policies is found here – MDC threat protection policies.

Microsoft Sentinel

Microsoft Sentinel has a native data connector for Azure Key Vaults that you stream Azure Key Vault diagnostics logs into the log repository. The connector uses Azure Policy to apply a single Azure Key Vault log-streaming configuration to a collection of instances, defined as a scope.

That being said, create Azure Policy and target it to the Key Vaults you want to have to send diagnostic logs to (data plane) Sentinel. I’m not covering security monitoring strategy here but I always pay attention to the collected data, collect only what you need based on threats. I’ve seen too many organizations that are collecting all possible audit data from different types of Azure resources and are now struggling with Sentinel costs.

Azure Log Analytics

Azure Log Analytics provides access to the raw data. If Azure Key Vault is configured to send diagnostic logs (data plane) data to the repository you can see both activities from both layers, management & data plane activities in LA.

Management plane activities in the AzureActivity table

AzureActivity
| where ResourceGroup contains "HONEYPOT-RG"
| where CategoryValue contains "Administrative" or CategoryValue contains "Security"
| extend message = tostring(parse_json(Properties).message)
| where ResourceProviderValue contains "MICROSOFT.KEYVAULT" or ResourceProviderValue contains "MICROSOFT.SECURITY"
| project TimeGenerated, Resource, ResourceGroup, OperationName, message, CategoryValue, Caller, CallerIpAddress

Data plane activities in the AzureActivity table filtered by sensitive operations

AzureDiagnostics 
| where TimeGenerated > ago(7d)
// Ignore Authentication operations with a 401. This is normal when using Key Vault SDK, first an unauthenticated request is done then the response is used for authentication.
| where Category == "AuditEvent" and not (OperationName == "Authentication" and httpStatusCode_d == 401)
| where OperationName in ('AfterAuthentication', 'CertificateCreate', 'CertificateGet', 'CertificateList', 'CertificateListVersions', 'CertificateGetDeleted' 'KeyCreate', 'KeyGet', 'KeyList', 'SecretGet', 'SecretList', 'SecretListDeleted', 'SecretListVersions', 'SecretSet', 'VaultGet', 'VaultPut')

Malicious Activity Simulation

To simulate malicious activity and detect the activity I used the following methods in my environment:

• Activity from the TOR browser to target Key Vault
• Used REST API & PowerShell and connected from another location (with the same & new access token)
• Used Managed Identity and called Azure Key Vault outside of Azure IPs – Kudos to Joosua Santasalo

With PowerShell

I used the PowerShell Az module for connecting and authenticating to Azure Key Vault, and read secrets from different locations around the globe. I also used the same access token from a different location (FI-BR) to simulate a token theft scenario.

With Rest API

Same as above, I called Azure Key Vault from multiple locations with the same user and simulated the token theft scenario.

Managed Identity to call Key Vault Outside of Azure IPs

In this scenario, with the help of Joosua, I created a VM and executed calls to Azure Key Vault & Azure Storage Account from different locations around the globe.

Detection

Defender for Cloud

As mentioned above, MDC detects suspicious activity mostly based on external threats. Here you can see alerts created from the simulated TOR activity. MDC has the following detection rules for identifying access from TOR exit nodes.

• Access from a suspicious IP address to a key vault
• Access from a TOR exit node to a key vault

The threat protection layer & detection rules for both Azure Key Vault & Azure Resource Manager created the following alerts:

• Access from a TOR exit node to a key vault
• Azure Resource Manager operation from suspicious proxy address

Key takeaway: Only access from the TOR network was detected. All the calls around the globe through PowerShell & REST API didn’t raise up alert.

Microsoft Sentinel & M365 Defender

On the Sentinel side, I’ve enabled built-in detection rules for Azure Key Vault and saw some incidents raised there based on the activities highlighted above. What kind of incidents you will see in Sentinel depends on your Sentinel analytics rule configuration, are you using built-in or custom rules.

For deeper investigation of identity-related activities, Sentinel UEBA & M365 UEBA provides useful data.

Key takeaway: Sentinel incidents are based on the analytics rules and I can see multiple incidents created by M365 Defender & Sentinel in my incident queue. Because of the native behavior of the logs, I’m not seeing a correlation between the IPs (& the solutions). Azure AD Managed Identity logs don’t contain location info. Malicious IPs are visible on AzureActivity & AzureDiagnostics table and also in the Azure AD Identity Protection (AADIP) detections..

Azure Log Analytics

Azure Log Analytics provides raw data from both, management & data plane IF diagnostics flow has been configured. To summarize:

• Management plane audit trail is enabled by default but you need to enable data ingestion in Azure subscription level
• The data plane audit trail is not enabled by default and you need to configure resource diagnostics settings separately

If prerequisites are met, you can see activities against your Key Vault in the Log Analytics.

This is where Joosua’s magic jump on the game. As said before, a custom KQL query is needed to see Managed Identity authentication traffic outside of the Azure IP ranges. During the demo session, Joosua showed me how we could trigger activities inside & outside of Azure IP ranges with Managed Identity against Key Vault & Storage account.

Tests showed some interesting information from the audit trail including activities against Key Vault & Azure AD Service Principal creation from the affected VMs.

The event in Azure AD Managed Identity Logs followed by malicious activity

Her you can find simulated malicious activity and the event in managed identity logs.

Using custom query for finding IPs where MI was used outside of Azure IP ranges with the custom query.

Key takeaway: There was a huge delay in the login events in Azure AD logs created by Managed Identity. I’m not sure if was it a ‘bad day’ for the audit trail or a coincidence but the delay was something around 10 hours. Correlation can be made with AzureActivity & AzureDiagnostics tables that contain IP information.

Summary & Caveats

• Azure Key Vault data plane activities are not monitored by default
• Follow Microsoft’s best practices to protect your Key Vaults from adversaries accessing them
  • Enable data plane auditing
  • Set up monitoring to detect suspicious activity.
    • Create detections for the use case mentioned in this blog (login outside Azure IPs)
  • Use RBAC to control access to the Key Vaults, it provides more granular access controls than policies
  • Use Azure Key Vault firewall and virtual network service endpoints You can use Azure Key Vault firewall and virtual network service endpoints to further secure access to Azure Key Vault
• If Managed HSM is the way to go take this guidance into account (Managed HSM Logging)
• Consider of isolating Key Vault
  • Integrate Key Vault with Azure Private Link
  • Policy for Azure Key Vault Private Link
  • Blog by Aidan Finn about How to detect connections to a PaaS resource using a Private Endpoint

• Key Vault firewalls and virtual network rules only apply to the data plane of Key Vault.
• Management control plane operations (such as creating access policies, setting access firewalls, etc) are not affected by firewalls and virtual network rules.
• Create custom queries for detecting activity outside Azure IP ranges (in Managed Identity case)
• Take into account possible delays in the AAD Managed Identity logs
• Take into account the native behavior of Azure AD logs (Managed Identity doesn’t include IP address)

In conclusion, auditing Azure Key Vault activities from malicious actors is critical to maintaining the security of your sensitive information.

Hope this helps, until next time!