Azure Key Vault is a cloud-based service that allows you to securely store and manage sensitive information such as passwords, certificates, and encryption keys. As with any system that stores sensitive information, it’s important to ensure that access to Azure Key Vault is properly controlled and monitored to prevent unauthorized access by malicious actors.
Lately, I had a case where I needed to dig deep into the Azure Key Vault audit trail for investigation purposes. In this post, I’ll cover how to audit Azure Key Vault for detecting activities from malicious actors with the tools I’ve found the most useful.
Before we dive into the details of Azure Key Vault, I’d like to give a big shout-out to Joosua Santasalo for his invaluable contributions to this topic.
It’s essential to understand how Azure resources are audited and what layers are included. My fellow MVP Joosua Santasalo created the following picture a few years ago and I still find it very useful for describing management & data plane differences in the Azure audit trail.
Management Plane Activities
The Azure management plane audit refers to the auditing of the Azure management plane, which is responsible for managing and controlling Azure resources, services, and operations. The management plane includes Azure Resource Manager, Azure Portal, Azure CLI, Azure PowerShell, and Azure API.
The Azure management plane audit is designed to provide visibility into activities and changes made to the management plane and to help identify potential security risks, compliance violations, and operational issues. It involves reviewing logs and audit trails generated by Azure services and tools that interact with the management plane and analyzing them to identify suspicious or anomalous activity.
The following activities are included in the Azure management plane audit:
- Authentication and authorization: This includes reviewing activities related to user authentication and authorization, such as changes to Role-based Access Control (RBAC) or changes to resource settings.
- Resource management: This includes reviewing activities related to the creation, modification, and deletion of Azure resources, such as virtual machines, storage accounts, and databases.
- Access control: This includes reviewing activities related to access control, such as changes to security groups, network security groups, and firewall rules.
- Monitoring and alerting: This includes reviewing activities related to monitoring and alerting, such as the configuration of Azure Monitor, alerts, and notifications.
- Configuration management: This includes reviewing activities related to the configuration of Azure services and tools, such as Azure policies, templates, and resource locks.
Overall, the Azure management plane audit is a critical component of Azure security and helps organizations maintain visibility and control over their Azure environment.
Note: Management plane activities are logged into the ‘AzureActivity’ table in Log Analytics.
Data plane activities
Refer to operations performed on the data stored in Azure services, such as Azure Key Vault or other Azure services. These activities involve accessing, modifying, or deleting data, and they are distinct from management plane activities that involve managing and controlling the Azure environment.
Similar to the Azure management plane audit, the Azure data plane audit is designed to provide visibility into data-related activities and to help identify potential security risks, compliance violations, and operational issues.
The following activities are typically included in the Azure data plane audit:
- Data access: This includes reviewing activities related to accessing data stored in Azure services, such as reading, writing, and copying data.
- Data modification: This includes reviewing activities related to modifying data stored in Azure services, such as adding, updating, or deleting data.
- Data retention and deletion: This includes reviewing activities related to data retention and deletion policies, such as configuring retention periods and deleting data that is no longer needed.
- Data backup and recovery: This includes reviewing activities related to backing up and restoring data, such as creating backups, scheduling backups, and recovering data from backups.
- Data encryption and security: This includes reviewing activities related to data encryption and security, such as configuring encryption settings, managing access control, and monitoring security logs.
Overall, the Azure data plane audit is important for maintaining the security, compliance, and operational effectiveness of Azure services that store and process data. By monitoring data-related activities and analyzing audit trails, organizations can identify potential issues and take appropriate actions to mitigate risks, and ensure the integrity and availability of their data.
Note: Data plane activities are logged into the ‘AzureDiagnostics’ table in Log Analytics.
Logs Available in Azure
Besides data & management plane there are other logs available in Azure. The following table lists most of them (source)
|Log category||Log type||Usage||Integration|
|Activity logs||Control-plane events on Azure Resource Manager resources||Process data/security alerts||REST API, Azure Monitor|
|Azure Resource logs||Frequent data about the operation of Azure Resource Manager resources in subscription||Provides insight into operations that your resource itself performed||Frequent data about the operation of Azure Resource Manager resources in the subscription|
|Azure Active Directory reporting||Logs and reports||Reports user sign-in activities and system activity information about users and group management||Microsoft Graph|
|Virtual machines and cloud services||Windows Event Log service and Linux Syslog||Captures system data and logging data on the virtual machines and transfers that data into a storage account of your choice||Windows (using Azure Diagnostics] storage) and Linux in Azure Monitor|
|Azure Storage Analytics||Storage logging, provides metrics data for a storage account||Provides insight into trace requests, analyzes usage trends, and diagnoses issues with your storage account||REST API or the client library|
|Network security group (NSG) flow logs||Storage logging provides metrics data for a storage account||Displays information about ingress and egress IP traffic through a Network Security Group||Azure Network Watcher|
|Application insight||Logs, exceptions, and custom diagnostics||Provides an application performance monitoring (APM) service for web developers on multiple platforms||REST API, Power BI|
|JSON format shows outbound and inbound flows on a per-rule basis||Microsoft Defender for Cloud alerts, Azure Monitor logs alerts||Provides security information and alerts||REST APIs, JSON|
Solutions for Security Monitoring
Microsoft Defender for Cloud (MDC)
MDC provides Cloud Workload Platform Protection (CWPP) layer for multiple services such as Azure Key Vault. I’m not covering the full Defender for Cloud enhanced protection layer here but rather focusing on Azure Key Vault protection instead.
What are the benefits of using MDC for Key Vault?
One of the key ways that Microsoft Defender for Cloud protects Azure Key Vaults is by providing continuous monitoring of Key Vault activities, and by using machine learning to detect suspicious or anomalous behavior. This includes monitoring for unauthorized access attempts, unusual login patterns, and other indicators of potential threats.
Additionally, Microsoft Defender for Cloud provides threat intelligence that helps to identify known threats and vulnerabilities that may affect Key Vaults. This includes leveraging the Microsoft Intelligent Security Graph, which aggregates threat data from multiple sources to provide a comprehensive view of the threat landscape.
MDC provides a wide range of Azure Policies for Key Vault posture management. The full list of available built-in policies is found in Azure Policies for Key Vault.
When you enabled Defender for Cloud protection for Azure Key Vault you will get protection mainly for external threats. The list of the Azure Keu Vault policies is found here – MDC threat protection policies.
Microsoft Sentinel has a native data connector for Azure Key Vaults that you stream Azure Key Vault diagnostics logs into the log repository. The connector uses Azure Policy to apply a single Azure Key Vault log-streaming configuration to a collection of instances, defined as a scope.
That being said, create Azure Policy and target it to the Key Vaults you want to have to send diagnostic logs to (data plane) Sentinel. I’m not covering security monitoring strategy here but I always pay attention to the collected data, collect only what you need based on threats. I’ve seen too many organizations that are collecting all possible audit data from different types of Azure resources and are now struggling with Sentinel costs.
Azure Log Analytics
Azure Log Analytics provides access to the raw data. If Azure Key Vault is configured to send diagnostic logs (data plane) data to the repository you can see both activities from both layers, management & data plane activities in LA.
Management plane activities in the AzureActivity table
AzureActivity | where ResourceGroup contains "HONEYPOT-RG" | where CategoryValue contains "Administrative" or CategoryValue contains "Security" | extend message = tostring(parse_json(Properties).message) | where ResourceProviderValue contains "MICROSOFT.KEYVAULT" or ResourceProviderValue contains "MICROSOFT.SECURITY" | project TimeGenerated, Resource, ResourceGroup, OperationName, message, CategoryValue, Caller, CallerIpAddress
Data plane activities in the AzureActivity table filtered by sensitive operations
AzureDiagnostics | where TimeGenerated > ago(7d) // Ignore Authentication operations with a 401. This is normal when using Key Vault SDK, first an unauthenticated request is done then the response is used for authentication. | where Category == "AuditEvent" and not (OperationName == "Authentication" and httpStatusCode_d == 401) | where OperationName in ('AfterAuthentication', 'CertificateCreate', 'CertificateGet', 'CertificateList', 'CertificateListVersions', 'CertificateGetDeleted' 'KeyCreate', 'KeyGet', 'KeyList', 'SecretGet', 'SecretList', 'SecretListDeleted', 'SecretListVersions', 'SecretSet', 'VaultGet', 'VaultPut')
Malicious Activity Simulation
To simulate malicious activity and detect the activity I used the following methods in my environment:
- Activity from the TOR browser to target Key Vault
- Used REST API & PowerShell and connected from another location (with the same & new access token)
- Used Managed Identity and called Azure Key Vault outside of Azure IPs – Kudos to Joosua Santasalo
I used the PowerShell Az module for connecting and authenticating to Azure Key Vault, and read secrets from different locations around the globe. I also used the same access token from a different location (FI-BR) to simulate a token theft scenario.
With Rest API
Same as above, I called Azure Key Vault from multiple locations with the same user and simulated the token theft scenario.
Managed Identity to call Key Vault Outside of Azure IPs
In this scenario, with the help of Joosua, I created a VM and executed calls to Azure Key Vault & Azure Storage Account from different locations around the globe.
Defender for Cloud
As mentioned above, MDC detects suspicious activity mostly based on external threats. Here you can see alerts created from the simulated TOR activity. MDC has the following detection rules for identifying access from TOR exit nodes.
- Access from a suspicious IP address to a key vault
- Access from a TOR exit node to a key vault
The threat protection layer & detection rules for both Azure Key Vault & Azure Resource Manager created the following alerts:
- Access from a TOR exit node to a key vault
- Azure Resource Manager operation from suspicious proxy address
Key takeaway: Only access from the TOR network was detected. All the calls around the globe through PowerShell & REST API didn’t raise up alert.
Microsoft Sentinel & M365 Defender
On the Sentinel side, I’ve enabled built-in detection rules for Azure Key Vault and saw some incidents raised there based on the activities highlighted above. What kind of incidents you will see in Sentinel depends on your Sentinel analytics rule configuration, are you using built-in or custom rules.
For deeper investigation of identity-related activities, Sentinel UEBA & M365 UEBA provides useful data.
Key takeaway: Sentinel incidents are based on the analytics rules and I can see multiple incidents created by M365 Defender & Sentinel in my incident queue. Because of the native behavior of the logs, I’m not seeing a correlation between the IPs (& the solutions). Azure AD Managed Identity logs don’t contain location info. Malicious IPs are visible on AzureActivity & AzureDiagnostics table and also in the Azure AD Identity Protection (AADIP) detections..
Azure Log Analytics
Azure Log Analytics provides raw data from both, management & data plane IF diagnostics flow has been configured. To summarize:
- Management plane audit trail is enabled by default but you need to enable data ingestion in Azure subscription level
- The data plane audit trail is not enabled by default and you need to configure resource diagnostics settings separately
If prerequisites are met, you can see activities against your Key Vault in the Log Analytics.
This is where Joosua’s magic jump on the game. As said before, a custom KQL query is needed to see Managed Identity authentication traffic outside of the Azure IP ranges. During the demo session, Joosua showed me how we could trigger activities inside & outside of Azure IP ranges with Managed Identity against Key Vault & Storage account.
Tests showed some interesting information from the audit trail including activities against Key Vault & Azure AD Service Principal creation from the affected VMs.
The event in Azure AD Managed Identity Logs followed by malicious activity
Her you can find simulated malicious activity and the event in managed identity logs.
Using custom query for finding IPs where MI was used outside of Azure IP ranges with the custom query.
Key takeaway: There was a huge delay in the login events in Azure AD logs created by Managed Identity. I’m not sure if was it a ‘bad day’ for the audit trail or a coincidence but the delay was something around 10 hours. Correlation can be made with AzureActivity & AzureDiagnostics tables that contain IP information.
Summary & Caveats
- Azure Key Vault data plane activities are not monitored by default
- Follow Microsoft’s best practices to protect your Key Vaults from adversaries accessing them
- Enable data plane auditing
- Set up monitoring to detect suspicious activity.
- Create detections for the use case mentioned in this blog (login outside Azure IPs)
- Use RBAC to control access to the Key Vaults, it provides more granular access controls than policies
- Use Azure Key Vault firewall and virtual network service endpoints You can use Azure Key Vault firewall and virtual network service endpoints to further secure access to Azure Key Vault
- If Managed HSM is the way to go take this guidance into account (Managed HSM Logging)
- Consider of isolating Key Vault
- Integrate Key Vault with Azure Private Link
- Policy for Azure Key Vault Private Link
- Blog by Aidan Finn about How to detect connections to a PaaS resource using a Private Endpoint
- Key Vault firewalls and virtual network rules only apply to the data plane of Key Vault.
- Management control plane operations (such as creating access policies, setting access firewalls, etc) are not affected by firewalls and virtual network rules.
- Create custom queries for detecting activity outside Azure IP ranges (in Managed Identity case)
- Take into account possible delays in the AAD Managed Identity logs
- Take into account the native behavior of Azure AD logs (Managed Identity doesn’t include IP address)
In conclusion, auditing Azure Key Vault activities from malicious actors is critical to maintaining the security of your sensitive information.
Hope this helps, until next time!