block AI training bots from crawling hosted websites

[Speartek]

We are using the Fortigate 200F running FortiOS v7.4.8 build2795 in front of our web hosting stack.  I want to block these AI training bots because they are not respecting robots.txt and using too much bandwidth.  I was thinking of using using a custom Application Signature to filter by the user-agent like "ClaudeBot" for example and then create a custom Application Control to block those requests.  I found a similar example but could not get it to work.  Below is the Application Signature I am using, any help would be much appreciated.  

F-SBID( --attack_id 9003; --name "ClaudeBot.Simple.Block"; --protocol tcp; --flow from_client; --seq =,1,relative; --data_size <0x500; --pattern "ClaudeBot/1.0|0d 0a|"; --context packet; --distance 0,context; --weight 20; --app_cat 28; )

[Anthony_E]

Hello,

Thank you for using the Community Forum. I will seek to get you an answer or help. We will reply to this thread with an update as soon as possible.

Thanks,

[Anthony_E]

Hello,

We are still looking for someone to help you.

We will come back to you ASAP.

Thanks,

[Anthony_E]

Hello,

To block AI training bots from crawling hosted websites using FortiGate, follow these steps:

1. Web Filter Profile Configuration: Access the FortiGate web-based manager.
2. Navigate to `Security Profiles` > `Web Filter`. 
3. Create or edit a web filter profile.
4. Block AI Category*: In the web filter profile, locate the `FortiGuard Categories`. - Find the `Artificial Intelligence Technology` category.
5. Set this category to `Block`.
6. Apply Web Filter Profile: Apply the configured web filter profile to the relevant firewall policy that controls access to your hosted websites.
7. Verify URL Category: Use the FortiGuard Web Filter Lookup tool (https://www.fortiguard.com/webfilter) to verify that the URLs you want to block are categorized under `Artificial Intelligence Technology`.
8. Monitor and Adjust: Monitor the logs to ensure that the AI training bots are being blocked.
9. Adjust the web filter settings as necessary to improve effectiveness.