A couple weeks ago I was asked by a client that could I make short research on Microsoft Defender for Endpoint (MDE) Web Content Filtering. I thought that the tasks would be much faster than it was. I quickly realized that there are multiple solutions in the game: Web Threat Protection, Web Content Filtering & Custom Indicator (+ Cloud App Security). In this blog post, you can find the results of the research. Worth mentioning that Web Content Filtering was announced to general availability mode last week.

Setting up the Scene

‘Web Content Filtering’ is part of ‘Web Protection’ provided by Microsoft Defender for Endpoint. The other components are ‘Web Threat Protection’ & ‘Custom indicators’. In short, with these solutions, the organization’s devices are protected against web threats and provide a way to regulate/block content that is restricted in the organization.

In summary, Web Protection is made up of the:

• Web Threat Protection = WT
• Web Content Filtering = WCF
• Custom Indicators = CI

• 

Web Threat Protection

Web Threat Protection description from docs.microsoft.com:

Web threat protection is part of Web protection in Defender for Endpoint. It uses network protection to secure your devices against web threats. By integrating with Microsoft Edge and popular third-party browsers like Chrome and Firefox, web threat protection stops web threats without a web proxy and can protect devices while they are away or on-premises.

Web threat protection stops access to phishing sites, malware vectors, exploit sites, untrusted or low-reputation sites, as well as sites that you have blocked in your custom indicator list.

I’m not covering full deployment details and how the service needs to be configured, all required information is found from the official documentation.

Web threat protection includes:

• Comprehensive visibility into web threats affecting the organization.
• Investigation capabilities over web-related threat activity through alerts and comprehensive profiles of URLs and the devices that access these URLs.
• A full set of security features that track general access trends to malicious and unwanted websites.

Web Content Filtering

Web Content Filtering description from docs.microsoft.com:

Web content filtering is part of the Web protection capabilities in Microsoft Defender for Endpoint. It enables organizations to track and regulate access to websites based on their content categories. Policies can be configured based on Defender for Endpoint device groups which provides more granularity for the scenarios.

Web content filtering is available on the major web browsers, with blocks performed by Windows Defender SmartScreen (Microsoft Edge) and Network Protection (Chrome, Firefox, Brave, and Opera). 

Web content filtering includes:

• Users are prevented from accessing websites in blocked categories, whether they are browsing on-premises or away.
• You can conveniently deploy varied policies to various sets of users using the device groups defined in the Microsoft Defender for Endpoint role-based access control settings.
• You can access web reports in the same central location, with visibility over actual blocks and web usage.

Prerequisites and configuration instructions are found here.

Custom Indicators

Custom Indicators description from docs.microsoft.com:

Defender for Endpoint can block what Microsoft deems as malicious IPs/URLs, through Windows Defender SmartScreen for Microsoft browsers, and through Network Protection for non-Microsoft browsers or calls made outside of a browser.

By creating indicators for IPs and URLs or domains, these can be blocked or allowed when needed. An alternative option is to warn users with a prompt if they open a risky app and provide even greater control with Cloud App Security integration to Shadow IT applications.

The custom indicator includes:

• Ability to create IP and URL-based indicators of compromise to protect your organization against threats.
• Investigation capabilities over activities related to your custom IP/URL profiles and the devices that access these URLs.
• The ability to create Allow, Block, and Warn policies for IPs and URLs.

Order of Presence

As you can see from the table below ‘Custom Indicator’ policy always wins. This means that CI can be used if exclusions are needed to allow some specific web site for device groups.

From docs.microsoft.com: It’s possible to override the blocked category in web content filtering to allow a single site by creating a custom indicator policy. The custom indicator policy will supersede the web content filtering policy when it’s applied to the device group in question.

Custom Indicator policy	Web threat policy	WCF policy	MCAS policy	Result
Allow	Block	Block	Block	Allow (Web protection override)
Allow	Allow	Block	Block	Allow (WCF exception)
Warn	Block	Block	Block	Warn (override)

Web Threat Protection Reports

Web Threat Protection provides useful reports based on web browsing security. You can expect to find the following reports from the security.microsoft.com portal reports section:

• Web threat detections over time
• Web activity summary
• Web threat summary
• Web content filtering summay
• Web activity by category

• 

• 

Detailed information is provided if you select the ‘details’ with domains and machine groups included

• 

• 

Test Drive – Demo Scenarios

Based on my client request we decided to test the following scenarios:

1. How Web Content Filtering works in Windows Server 2012 R2 (with MDE:n new preview-version), and how in general works with server OS?
2. Is it possible to create exclusions to content filtering (for example using custom indicators)?
3. How Web Threat Protection reacts when file is downloaded from malicious URL with other manners/tools than browser)?
4. Order of presence in Web Threat Protection

Worth mentioning that in the demo environment there are two device groups: ‘Feta Servers’ & ‘Feta-ws’.

• 

1. How Web Content Filtering works Windows 10 and, does it work with server OS (especially with MDE:n new preview version agent)?

Based on the official documentation Web Protection, Network Protection & Custom Indicators are supported by:

• Windows 10 & Windows 11
• Windows Server 2012 R2
• Windows Server 2016
• Windows Server 2019 & 2022
• Windows Server 1803+

In the pictures below:

• The first one is blocked by Web Content Filtering (Smart Screen) based on blocked category ‘gambling’
• The second one is blocked by Web Content Filtering (Network Protection) on 3rd party browser by same category

• 

• 

An important factor is that based on the documentation, with Server OS Web Content Filtering works but only with Edge (cmd – systeminfo – OS configuration = Server). Detailed explanation from docs.microsoft.com:

Network Protection is only supported in Inspect mode on Server devices, which is responsible for securing traffic across supported third-party browsers.

Network Protection does not currently support SSL inspection, which may result in some sites being allowed by Web Content Filtering that would normally be blocked. Sites would be allowed due to a lack of visibility into encrypted traffic after the TLS handshake has taken place and an inability to parse certain redirects. This includes redirections from some web-based mail login pages to the mailbox page. As an accepted workaround, you can create a custom block indicator for the login page to ensure no users are able to access the site. Keep in mind, this may block their access to other services associated with the same website.

Based on how Network Protection (NP) works with 3rd party browsers (description above) if there is a redirection NP might not able to block access to the site. One good example is gmail.com where the redirect is from gmail.com to http://www.google.com/gmail. So be aware that all of the web requests invoked from PowerShell might not be necessarily blocked even though your policies wouldn’t allow connection to the sites.

• 

Windows Server 2019 – Build 17763 (1809)

Tests run with Server 2019 with the following sites:

• Gmail.com – redirect
• Mega.io – no redirect

Mega.io is found from the ‘Custom Indicators’ list and is blocked for that reason. Gmail.com is found from Web Content Filtering category ‘Web-based email’.

• 

• 

• 

Windows Server 2012 R2

All blocked sites were on the ‘Custom Indicators’ side. When the site was in the Web Content Filtering category it was not blocked on the sites tested.

• 

• 

• 

2. Is it possible to create exclusions to content filtering (for example using custom indicators)

The short answer is – Yes, it is. The Custom Indicators can be used to override policies from WT & WCF but take into account that ‘custom indicators’ can take 30-120min to be effective. Below you can see that Twitter is blocked from servers but allowed from W10 devices.

• 

3. How Web Threat Protection reacts when file is downloaded from malicious URL with other manners/tools than browser

In general, the file download is blocked if the URL is not allowed. Using custom indicators blocks access to the URL. Actual blocking is made by the Network Protection engine and ExploitGuard.

• 

If ‘Soft block’ aka warning is used the end-user experience is almost identical but in the toast message, you can see the ‘Unblock’ option. The same end result can be achieved if applications are managed from Microsoft Cloud App Security (MCAS)(sanctioned/unsanctioned). More information on how to manage Shadow IT with MCAS from my TechCommunity blog.

• 

4. Order of presence

The order of precedence relates to the order of operations by which a URL or IP is evaluated. For example, if you have a web content filtering policy you can create exclusions through custom IP/URL indicators.

• 

Threat Hunting

Here are some useful queries to find what component made the actual blocking, is the connection from 1p or 3rd browser or process related to Web Content Filtering & Custom Indicators.

//SmartScreenBlocks or ExploitGuardNetworkProtectionBlocked which is 3rd party process or browser
DeviceEvents

| where ActionType in ('SmartScreenUrlWarning','ExploitGuardNetworkProtectionBlocked')
| extend ParsedFields=parse_json(AdditionalFields)
| project DeviceName, ActionType, Timestamp, RemoteUrl, InitiatingProcessAccountName, ResponseCategory=tostring(ParsedFields.ResponseCategory),Experience=tostring(ParsedFields.Experience)
| where ResponseCategory == "Malicious" or Experience == "Malicious"

//Web Content Filtering - SmartScreenBlocks or ExploitGuardNetworkProtectionBlocked where responseCategory = 3rd party & Experience = 1p browser

DeviceEvents
| where ActionType in ('SmartScreenUrlWarning','ExploitGuardNetworkProtectionBlocked')
| extend ParsedFields=parse_json(AdditionalFields)
| project DeviceName, ActionType, Timestamp, RemoteUrl, InitiatingProcessFileName, ResponseCategory=tostring(ParsedFields.ResponseCategory), Experience=tostring(ParsedFields.Experience)
| where ResponseCategory == "CustomPolicy" or Experience == "CustomPolicy"

// Web Content Filtering - CustomBlockList Used


DeviceEvents
| where ActionType in ('SmartScreenUrlWarning','ExploitGuardNetworkProtectionBlocked')
| extend ParsedFields=parse_json(AdditionalFields)
| project DeviceName, ActionType, Timestamp, RemoteUrl, InitiatingProcessFileName, ResponseCategory=tostring(ParsedFields.ResponseCategory), Experience=tostring(ParsedFields.Experience)
| where ResponseCategory == "CustomBlockList" or Experience == "CustomBlockList"

Alerts

When malicious activity is detected Microsoft Defender for Endpoint creates an alert that is seen in M365 Defender. Depending on is alert created by block or audit action the name of the alert varies.

• 

• 

• 

• 

Links

Web Threat Protection

Web Content Filtering

Custom Indicators

Announcement of Web Content Filtering generally available