Troubleshooting Tip: Unable to block specific part of the website even with deep inspection enabled
Description
This article describes how to determine the URL to block when blocking a specific part of a website. Modern websites use JavaScript to send a HTTP request to the server without reloading the whole page. It will use the API fetch functions or XMLHttpRequest(xhr)/AJAX and can change the content of the website based on the response.
Scope
FortiGate.
Solution
Network Topology:
PC -> FortiGate -> Internet.
FortiGate Configuration:
Web Filter:
 
In this example, the URL is www.samsclub.com/c/clothing-shoes-accessories/1959?xid=hdr:shop:more-departments:clothing-shoes-accessories.
Firewall Policy:
 
- Access the website samsclub.com -> Clothing, Shoes & Accessories. The website will still be accessible.
 
- After reloading the page, the block replacement message is shown. In this scenario, it should be blocked upon selecting 'Clothing, Shoes & Accessories'.
 
- Open the browser Developer Tools -> Network. Ensure Preserve Log and Disable cache are checked. Access the website again -> select the Clear icon on the DevTools, then select Fetch/XHR to filter it.
 
- Look for the fetch or xhr that triggered when the 'Clothing, Shoes and Accessories' was selected.
- It is possible to test if this is the actual URL without blocking it yet on FortiGate. Right click -> Block request URL. Access the site again - the specific content of the website will be blocked. Clear all of the blocking after the test.
 
 
 
- Block the URL with static URL Filter on FortiGate. The specific content of the website should be blocked.
 
In this case, the URL is: www.samsclub.com/api/node/vivaldi/browse/v2/category/1959.
 
Related article:
Technical Tip: Using a static URL filter feature to allow/block web sites