FortiGate Next Generation Firewall utilizes purpose-built security processors and threat intelligence security services from FortiGuard labs to deliver top-rated protection and high performance, including encrypted traffic.
Article Id 189408
This article explains how to create a custom signature to block files according to the hash value of the file.

In this example, a custom signature will be created that allows a hash value (or checksum) to be specified of a file that is to be blocked.  To block multiple files, create a custom signature for each file with that file's hash value in it and then add all of the custom signatures to an IPS sensor and set the action to block for each one.  When IPS encounters a file with a matching hash value the file is blocked.

This example uses a CRC32 checksum of the file as the hash value of the file to be blocked.  Any utility can be used that supports CRC32 checksums to generate the hash value.

Enter the custom signature basic format.
All custom signatures have a header and at least one keyword/value pair. The header is always the same:
The keyword/value pairs appear within the parentheses and each pair is followed by a semicolon.

Choose a name for the custom signature
Every custom signature requires a name, so it is a good practice to assign a name before adding any other keywords.  Use the --name keyword to assign the custom signature a name.  The name value follows the keyword after a space.  Enclose the name value in double quotes:

F-SBID( --name "File.Hash.Example"; )

The signature, as it appears here, will not do anything if used.  It has a name, but does not look for any patterns in network traffic.

Specify the traffic type.
Use the --protocol tcp keyword to limit the effect of the custom signature to only TCP traffic.  This will save system resources by not unnecessarily scanning UDP and ICMP traffic.

F-SBID( --name "File.Hash.Example"; --protocol tcp; )

The FortiGate unit will limit its search for the pattern to TCP traffic and ignore UDP and ICMP network traffic.

Add the CRC32 hash value.
Use the --crc32 keyword.  This indicates that the value that follows is a hexadecimal number that represents the CRC32 checksum of the file.  The --crc32 keyword also requires that the file length is included.  The syntax is --crc32 <checksum>,<file-length>;.  The following example shows the syntax for a file with checksum 51480492 and file length 822.

F-SBID( --name "File.Hash.Example"; --protocol tcp; --crc32 51480492,822; )