This is a second part of the multi-cloud security & posture management series. The first one, high-level overview, is found from the following link:
In this post, I’ll cover both, CloudTrail (raw data) & GuardDuty (findings) integrations to security monitoring solutions, Azure Sentinel & Cloud App Security (MCAS). Security posture management deserves own post and it will be covered on later chapter in this series.
Special thanks to my colleague Panu Mälkki who’s extremely deep knowledge from AWS was crucial when building the integrations.
Architecture
Base of the architecture is one example case I have been working with. The idea in this architecture is to leverage Microsoft security solutions as much as possible in security monitoring & posture management. In the picture below, there are four (4) major log sources sending audit trail (alerts, data & signal, events / raw logs, recommendations, etc) to Microsoft security solutions (Azure Sentinel, Azure Security Center & Cloud App Security).
- Azure & M365
- D365
- Amazon Web Services (AWS)
- Google Cloud Platform (GCP)
- On-premises environment
Azure Sentinel
Azure Sentinel has built-in connector that can be used to ingest data from AWS CloudTrail. The connection process delegates access for Azure Sentinel to the AWS resource logs, creating a trust relationship between AWS CloudTrail and Azure Sentinel.
With AWS CloudTrail, you can monitor your AWS deployments in the cloud by getting a history of AWS API calls for your account, including API calls made via the AWS Management Console, the AWS SDKs, the command line tools, and higher-level AWS services. You can also identify which users and accounts called AWS APIs for services that support CloudTrail, the source IP address the calls were made from, and when the calls occurred.
Options for Integration
To ingest AWS data, there is a couple of alternative approaches to establish the integration:
Using Azure Sentinel native connector
- The first one is to use Azure Sentinel native connector and ingest AWS CloudTrail raw data to Azure Sentinel.
Using custom connector (Azure Function)
- The second one is to do some magic in the AWS side, send all findings from AWS GuardDuty to S3 bucket (while using multiple AWS accounts this can and should be a centralized bucket) and ingest only findings to Azure Sentinel via Azure Function that acts as a connector.
- Nothing prevents you from sending other data, such as CloudTrail events, to S3 bucket and collect the data from there but connector tuning might be needed (I haven’t test that scenario)
Using Logstash to integrated GuardDuty to Azure Sentinel (description from Mikko’s blog)
- The idea is to utilize AWS native Export GuardDuty Findings to S3 functionality, which enables GuardDuty to save findings in JSON format to an S3 bucket.
- Reading JSON data from S3 to Azure Sentinel is easy to do with Logstash, without being forced to build any custom translation patterns to map the data. Logstash also gives you the power to do filtering, mutation and enrichment if you want to.
More information from these approaches are found from the links below:
- Connect AWS CloudTrail to Azure Sentinel
- Ingesting AWS Guard Duty findings to Azure Sentinel written by Marko Lauren from Microsoft.
- Integrating AWS GuardDuty to Azure Sentinel with Logstash written by Mikko Koivunen.
Security Monitoring
Depending on selected approach you can expect to find AWS audit data from underlying Azure Log Analytics workspace. If data is ingested through native connector it will be found from AWSCloudTrail table. If you ingest data trough Azure Function data is ingested to the table defined in the Azure Function configuration.
The main difference between these approaches is the data that’s being ingested. The first one gives you raw data from CloudTrail, the second one GuardDuty findings.
Pros/cons with raw data:
- All raw data is in the same place (centralized view) and gives you possibility to handle security monitoring rules, threat hunting etc in the same place
- Costs increases because you will pay twice for storing the same data
- Responsibility areas
Pros/cons with GuardDuty findings:
- Raw data is only in AWS side and costs are lower
- Findings are sent to Azure Sentinel and actual investigation will be done in AWS side
- If SOC doesn’t have visibility/responsibility to AWS (it’s on different service provider) it could be easier to ingest only findings
- Raw data is missing, all monitoring capabilities are done in AWS side
- Raw data is missing, threat hunting needs to be done in AWS side unless done with external data operator (Azure Sentinel Notebooks / Log Analytics)
Analytic Rules for AWS
Azure Sentinel offers the following analytics rules by out of the box:
Threat Hunting for AWS
To use Azure Sentinel threat hunting you need to ingest raw data to underlying Azure Log Analytics workspace. Microsoft community has wide range of built-in hunting queries implemented to Azure Sentinel which can be used to get started. For AWS, the following queries are available for now when “CloudTrail” is selected as required source.
Caveats
- Custom connector support only scenario where IAM user is created on same AWS account with S3 log bucket.
- Pay attention of monitoring custom connector. Because it’s Azure Function, the logic is a bit different.
- Custom connector has built-in delta mechanism to avoid fetching same information twice. This seems to be based on timestamps only. If, for whatever reason, the connector can’t reach the source bucket you might loose some data (i.e. there is unetched data “between” deltas).
General guidance ‘How to monitor Azure Sentinel data connectors‘
Monitoring can be done by sending diagnostics logs to Log Analytics or straight from the console when needed.
Microsoft Cloud App Security (MCAS)
These policies look for activities within a single session with respect to the baseline learned, which could indicate a breach attempt. These detections leverage a machine learning algorithm that profiles the users log on a pattern and reduces false positives. These detections are part of the heuristic anomaly detection engine that profiles your environment and triggers alerts with respect to a baseline that was learned on your organization’s activity.
AWS Integration
Cloud App Security integration leverages AWS native security components capabilities using the connector API and gives the customer insights visibility from AWS to Microsoft security tools.
The following AWS connections are available in Microsoft Cloud App Security:
- Security auditing: Gives visibility into and control over AWS app use
- Requirement: AWS CloudTrail & CloudWatch
- Security configuration: Connection gives fundamental security recommendations based on the ‘Center for Internet Security (CIS)’ benchmark for AWS.
- Requirement: AWS Security Hub
Security Monitoring
When connecting app through ‘App Connector’ such as ‘Amazon Web Services’, my experience is that there might be delays (24-48 hours) depending on data source before data is seen in MCAS instance. Last time I connected AWS it took approximately 24 hours before anything was seen in MCAS portal.
From the docs.microsoft.com
- “Connection may take some time depending on the size of the tenant, the number of users, and the size and number of files that need to be scanned“
- Taking into account different limitations services impose on the APIs, the Cloud App Security engines use the allowed capacity. Some operations, such as scanning all files in the tenant, require numerous APIs so they’re spread over a longer period. Expect some policies to run for several hours or several days.
When connection is established you have raw data (such as account activities & API calls) and you can start creating custom rules based on your use cases.
Benefit for connecting AWS to MCAS:
- Cloud Security Posture management
- Compromised account or insider threat
- Data leakage protection
When AWS integration in MCAS is enabled these policies are automatically enabled in monitoring mode, no governance actions are enforced.
MCAS offers the following built-in templates to detect and notify potential threats. New detections are added on a regular basis, I encourage to check policy updates from the reference link list. I would start with the built-in policies, but as said, custom cases can be created based on the ingested raw data.
| Type | Name |
| Activity policy template | Admin console sign-in failures CloudTrail configuration changes EC2 instance configuration changes IAM policy changes Logon from a risky IP address Network access control list (ACL) changes Network gateway changes S3 configuration changes Security group configuration changes Virtual private network changes |
| Built-in anomaly detection policy | Activity from anonymous IP addresses Activity from infrequent country Activity from suspicious IP addresses Impossible travel Activity performed by terminated user (requires AAD as IdP) Multiple failed login attempts Unusual administrative activities Unusual multiple storage deletion activities (preview) Multiple delete VM activities Unusual multiple VM creation activities (preview) |
| File policy template | S3 bucket is publicly available |
Security Posture Management in Cloud App Security
Even though security posture management gives crucial information from the environment configuration state the MCAS feature is overlapping with Azure Security Center “Cloud Security Posture Management – CSPM”. In my opinion, this integration is not needed in most of the environments / clients I work with. My recommendation is to use Azure Security Center instead which gives better visibility for CSPM than MCAS. There might be some use cases that I’m not aware of where this integration could be useful.
If you want to configure security posture management in MCAS you need to add it from “Connected Apps – Security Configuration Apps”. I have connections to two AWS instances, 1 GCP and 1 Azure. It has different pre-requisites than CloudTrail events integration, found from here.
Caveats
- To create “Activity policy” in MCAS AWS app connector needs to be in place
- Alerts are based on data received from AWS API – not real-time detections
- To create “File policy with governance action” you need to have MCAS behaving as a proxy (Azure AD + Conditional Access) – session control policies
- AWS CloudTrail has built-in limitations in its LookupEvents API
- It allows no more than two transactions per second (TPS) per account, and each query can return a maximum of 50 records
- Consequently, if a single tenant constantly generates more than 100 records per second in one region, backlogs and delays in data ingestion will result
- Azure Sentinel collects CloudTrail management events from all regions. It is recommended that you do not stream events from one region to another
- AWS custom connector support only scenario where IAM user is created on same AWS account with S3 log bucket.
- Custom connector has built-in delta mechanism to avoid fetching same information twice. This seems to be based on timestamps only. If, for whatever reason, the connector can’t reach the source bucket you might loose some data (i.e. there is unetched data “between” deltas).
- If the ‘Security Configuration‘ integration is needed, remember to enable AWS Security Hub before enabling integration. It’s separate from CloudTrail & CloudWatch integration which feeds data to the Activity Log.
Summary
In general, data ingestion to Azure Sentinel and other Microsoft security tools is quite straightforward to establish. As mentioned in this blog post, there are different approaches depending on organization needs, service provider(s) responsibilities, SLA’s and so on.
I have seen both described approaches (raw data & findings) in cases I have worked with. Personally, I like the idea of bringing only GuardDuty findings from AWS to Azure Sentinel. Main reasons are costs and the idea bringing only data what’s needed from other clouds to build effective security monitoring environment. Of course, reason vary and there isn’t a silver bullet for multi-cloud security monitoring.
In the next post I’ll cover how you can ingest Google Cloud Platform audit data to Microsoft security solutions before going into security posture management.
References
https://docs.microsoft.com/en-us/azure/sentinel/connect-aws
https://github.com/Azure/Azure-Sentinel/tree/master/DataConnectors/AWS-S3-AzureFunction
Sam's Corner 