|
Introduction.
The FortiGate application control feature has a wide range of standard application detection signatures that can be used to create firewall policies to granularly filter traffic. In addition to that, FortiGate also supports writing custom application signatures which can be used to match very specific and customized patterns of an application flow, based on client characteristics, traffic protocol type as well as server characteristics. This article describes how to do an in-depth troubleshooting of custom signatures when they are not matching the traffic pattern that is intended after configuration, and tune the signatures based on the debugging information.
Configurations.
The following is an example custom signature that will be used as a reference for the remainder of this article, with a signature to block application flows to LLM servers if the client requests are coming from 'curl'. The application signature would parse the HTTP headers (after decryption with deep inspection) and look for the client header fields containing the keyword curl, and if present, the session would be blocked.
FortiGate # show application custom
config application custom
edit "curl-block"
set comment ''
set signature "F-SBID( --attack_id 6485; --name \"curl-clients-block\"; --service HTTP; --protocol tcp; --app_cat 25; --flow from_client; --pattern \"curl\"; --no_case; --context header; --weight 40; )"
set category 25
next
end
FortiGate # show application list
config application list
. . .
edit "curl-LLM-block"
set other-application-log enable
config entries
edit 1
set application 6485 <------------ "Custom app signature blocking curl based access to applications"
set action block
next
edit 2
set category 36 <----------- "Category 36 maps to Generative AI applications"
set action pass
end
next
end
FortiGate # show firewall policy
config firewall policy
edit 2
set name "Allow-LLM-except-for-curl-clients"
set uuid 15ff17cc-dc51-51f0-7688-a38b5e13984e
set srcintf "port2"
set dstintf "port1"
set action accept
set srcaddr "Internal-Network"
set dstaddr "all"
set schedule "always"
set service "ALL"
set utm-status enable
set ssl-ssh-profile "deep-inspection"
set application-list "curl-LLM-block"
set logtraffic all
set logtraffic-start enable
set nat enable
next
end
If everything is good with the signature and policy configs, and the pattern match were to happen correctly, then a client trying to access an LLM application will be blocked with a message. But an HTTPS browser based connection to the same destination would be allowed based on the configs.
Linux# curl -k https://chatgpt.com
Output for the above command on the client machine would show the 'Application Blocked' message below and block the connection.
FortiGate will record a log event for this block action as shown below.
date=2025-12-18 time=12:48:33 eventtime=1766090912867448293 tz="-0800" logid="1059028705" type="utm" subtype="app-ctrl" eventtype="signature" level="warning" vd="root" appid=6485 srcip=192.168.20.1 srccountry="Netherlands" dstip=A.B.C.D dstcountry="United States" srcport=46700 dstport=443 srcintf="port2" srcintfrole="undefined" dstintf="port1" dstintfrole="undefined" proto=6 service="HTTPS" direction="outgoing" policyid=2 poluuid="15ff17cc-dc51-51f0-7688-a38b5e13984e" policytype="policy" sessionid=2188826 applist="curl-LLM-block" action="block" appcat="Web.Client" app="curl-clients-block" hostname="chatgpt.com" incidentserialno=110123165 url="/" agent="curl/7.81.0" httpmethod="GET" msg="Web.Client: curl-clients-block"
Debugging steps.
If the application signature is not working as intended and not blocking the expected application traffic, it could be tricky initially without additional debugging to find out why the configured signature is not working as expected. Use the following steps to debug further, note the caution while running these steps.
Caution: Enabling the full ips debug on a production FortiGate (even with filters enabled) is not recommended. These debugs are to be run either on test devices or during non-business hours during change windows, as the IPS debugs would cause 100% CPU usage.
Step 1: Ensure SSL deep inspection is enabled.
If deep inspection is not enabled, the application control feature would not be able to read the HTTP headers and the custom signature would not be able to match.
Step 2: Use flow mode to show the full IPS debug.
Flow mode shows the detailed IPS debug logs while the proxy mode does not. So, if proxy mode is configured for the firewall policy, either change it to flow mode briefly, or clone and create a new firewall policy with the only change of inspection mode on this policy entry from proxy to flow mode.
Step 3: SSH access or console to FortiGate for CLI.
Access the FortiGate CLI via SSH or console with putty or similar terminal client, and enable logging of console output to a file.
Step 4: Close the current application session before starting debug log capture.
To avoid any cached flow affecting the analysis, before starting the debug log capture, make sure the application is completely closed.
Step 5: Enable IPS debugs.
Enter the following command to set the engine count to 1.
config ips global
set engine-count
end
Note: CPU is expected to spike up to 100% while running IPS debugs, enable below debugs only during non-production hours, or better yet, run it only on test machines to validate the custom application signature. Ensure at least two filters are set to make the debugging very specific. In this example, filters are set for the IP address of the client and the port in use for the application traffic
Enter the following CLI commands to start capturing the IPS debug log.
diagnose ips filter set "host xxx.xxx.xxx.xxx"
diagnose ips filter set "port 443"
diagnose ips debug enable all
diagnose debug enable
## Replace the xxx.xxx.xxx.xxx with the IP address of the client device.
Step 6: Start the application access from the client machine.
Monitor the debugs on the CLI screen and after a sufficient amount of debugs are collected, move to the next step
Step 7: Stop the IPS debugs.
After the debugs are collected, enter the following CLI commands to stop the debug log capture:
diagnose debug disable
diagnose ips debug disable all
Step 8: Change the engine count back to default.
Enter the following command to set the engine count back to default.
config ips global
set engine-count 0
end
Note: If engine-count is set to the default value of 0 (which is what it should be under normal conditions), FortiOS sets the number to optimize performance depending on the number of CPU cores. So always keep it at the default 0, unless during troubleshooting.
Debug Analysis.
Here is an example analysis of the IPS debugs, that illustrates the specific sections of the debugs that are relevant for troubleshooting.
FortiGate # config ips global
FortiGate (global) # set engine-count 1
FortiGate (global) # end
FortiGate #
FortiGate # diagnose ips filter set "host 192.168.20.1"
FortiGate # diagnose ips filter set "port 443"
FortiGate # diagnose ips debug enable all
FortiGate # diagnose debug enable
FortiGate #
Challenger-kvm47 # [6431@-1]ips_tcp_layer_fini: sess: 37872, tcp fini called
[6431@-1]ips_tcp_layer_fini: sess: 37874, tcp fini called
[6431@-1]ips_run_packet_prepare: got a packet, id=91141, size=60
[6431@-1]ips_process_event: ctx 0: 5 => 0
<snippet>
PACKET id:91141 len:60 vf:0 vrf:0 fw:2 view:3 derived:0 encap:0 log:(traffic:0 pre:1 post:0)
imp2p:0x0 proxy:0x0 features:0x4 flowutm:1 input:raw <----------- "Incoming application traffic"
192.168.20.1:46700 -> A.B.C.D:443 protocol:6
IP length:60b, header:20b, ttl:63, tos:0, id:52180
TCP payload:0b, header:40b
TCP seq:878960204, ack:0, win:64240, flags:******S*
[6431@-1]ips_run_decode: ips_pkt_id: 91141 <---------- "First packet being procssed by IPS engine"
0000 45 00 00 3C CB D4 40 00 3F 06 81 80 53 53 53 02 E..<..@.?...SSS.
0010 AC 40 9B D1 B6 6C 01 BB 34 63 DE 4B 00 00 00 00 .@...l..4c.K....
0020 A0 02 FA F0 EA D6 00 00 02 04 05 B4 04 02 08 0A ................
0030 7C 51 2C A9 00 00 00 00 01 03 03 07 |Q,.........
[6431@-1]ips_run_session_verdict_check: can't find session
[6431@-1]ips_create_tcp_session: SYN packet from client
[6431@-1]ips_create_session: enter <------------------ "Creating a new sessions"
[6431@-1]ips_scan_range_get_range: found in cache, view:3 proto:6 range:25600
[6431@-1]ips_create_session: set ignore_app_after_size from 204800 to 25600 by dependencies of 0 Root
[6431@-1]ips_sess_get_dev_idx: sif = 4, dif = 3
[6431@37895]ips_asm_set_seq: set asm seq: 3463de4c
[6431@37895]ips_transit_tcp_state: (C:SYN_SENT S:LISTEN) -- SYN ->
[6431@37895]ips_handle_tcp_action: (C:SYN_SENT S:SYN_RCVD) act=FIRST_CLIENT_PKT
[6431@37895]ips_l7_dsct_processor: serial=2188826 create: ssl
[6431@37895]ips_l7_dsct_processor: serial=2188826 create: http
[6431@37895]ips_l7_dsct_processor: serial=2188826 create: http2
<snippet>
[6431@37896]ips_run_decode: ips_pkt_id: 91160 <---------- "Subsequent packets being processed by IPS engine"
0000 45 00 00 5D CB E0 40 00 3F 06 81 53 53 53 53 02 E..]..@.?..SSSS.
0010 AC 40 9B D1 B6 6C 01 BB 34 63 E1 23 D9 79 E7 36 .@...l..4c.#.y.6
0020 50 18 01 F5 00 00 00 00 00 00 2C 01 05 00 00 00 P.........,.....
0030 01 82 84 87 41 88 24 E3 4C D5 A5 72 1E 9F 7A 88 ....A.$.L..r..z.
0040 25 B6 50 C3 AB BC 15 C1 53 03 2A 2F 2A 40 89 F2 %.P.....S.*/ *@..
0050 B1 2D 42 4F 4A C9 1C DF 84 08 99 69 BF .-BOJ......i.
[6431@37896]ips_run_session_verdict_check: serial=2188826 session is ACTIVE <------- "Packets recognized to be part of an existing active session"
[6431@37896]ips_dsct_session_loop: serial=2188826 only: http2
[6431@37896]ips_http2_on_pkt: enter
[6431@37896]http2_data_process: sess 37896, from 1, on stream 1, type header, flags 05, len 44
[6431@37896]alloc_stream: sess 37896, create stream 1, type 1, flags 5, len 44
[6431@37896]http2_data_process: recv clt FLAG_END_STREAM, sess_id=37896, pkt_id=91160
[6431@37896]http2_data_process: sess 37896, stream 1, avail 0, input 44, total 44
[6431@37896]get_index_len: inflatehd: indexed repr
[6431@37896]get_index_len: prefix 7, opcode 1, index_len 1
[6431@37896]on_header: name: :method, value: GET
[6431@37896]get_index_len: inflatehd: indexed repr
[6431@37896]get_index_len: prefix 7, opcode 1, index_len 1
[6431@37896]on_header: name: :path, value: /
[6431@37896]get_index_len: inflatehd: indexed repr
[6431@37896]get_index_len: prefix 7, opcode 1, index_len 1
[6431@37896]on_header: name: :scheme, value: https
[6431@37896]get_index_len: inflatehd: literal header repr - indexed name
[6431@37896]get_index_len: inflatehd: indexing required=1, no_index=0
[6431@37896]get_index_len: prefix 6, opcode 3, index_len 1
[6431@37896]on_header: name: :authority, value: chatgpt.com
[6431@37896]get_index_len: inflatehd: literal header repr - indexed name
[6431@37896]get_index_len: inflatehd: indexing required=1, no_index=0
[6431@37896]get_index_len: prefix 6, opcode 3, index_len 1
[6431@37896]on_header: name: user-agent, value: curl/7.81.0 <--------- "user-agent tag and value confirming curl"
[6431@37896]ips_http_user_agent_offset: user-agent (others) ua_offset=0: curl/7.81.0
[6431@37896]get_index_len: inflatehd: literal header repr - indexed name
[6431@37896]get_index_len: inflatehd: indexing required=1, no_index=0
[6431@37896]get_index_len: prefix 6, opcode 3, index_len 1
[6431@37896]on_header: name: accept, value: */ *
[6431@37896]get_index_len: inflatehd: literal header repr - new name
[6431@37896]get_index_len: inflatehd: indexing required=1, no_index=0
[6431@37896]get_index_len: prefix 6, opcode 2, index_len 1
[6431@37896]on_header: name: x-custom-tag, value: 12345
[6431@37896]http2_data_process: sess 37896, stream 1 frame data complete
[6431@37896]update_field_offset: offset=0 delta=14 total=118
[6431@37896]assign_header: host: chatgpt.com <--------- "Matching the LLM server fqdn"
[6431@37896]assign_header: URL: GET / HTTP/2
DECODED:
-------------------------HTTP2 Headers-------------------------
0000 47 45 54 20 2F 20 48 54 54 50 2F 32 0D 0A 3A 70 GET / HTTP/2..:p <------ "Review the decoded header to see the header tags are correctly listed"
0010 61 74 68 3A 20 2F 0D 0A 3A 73 63 68 65 6D 65 3A ath: /..:scheme:
0020 20 68 74 74 70 73 0D 0A 48 6F 73 74 3A 20 63 68 https..Host: ch
0030 61 74 67 70 74 2E 63 6F 6D 0D 0A 75 73 65 72 2D atgpt.com..user-
0040 61 67 65 6E 74 3A 20 63 75 72 6C 2F 37 2E 38 31 agent: curl/7.81 <------ "curl tag in the user-agent field, look for exact tag:value match"
0050 2E 30 0D 0A 61 63 63 65 70 74 3A 20 2A 2F 2A 0D .0..accept: */ *.
0060 0A 78 2D 63 75 73 74 6F 6D 2D 74 61 67 3A 20 31 .x-custom-tag: 1
0070 32 33 34 35 0D 0A 2345..
[6431@37896]ips_http2_get_referer_by_session: found stream 1 in get http2 referer
[6431@37896]ips_process_event: ctx 0: 0 => 1
[6431@37896]ips_process_event: ctx 0: 1 => 1
[6431@37896]ips_process_event: ctx 0: 1 => 1
[6431@37896]ipsa_adapter_search_prepare: IPSA is disabled
[6431@37896]ips_process_event: ctx 0: 1 => 2
[6431@37896]ips_process_event: ctx 0: 2 => 4
[6431@37896]ips_match_rule: pattern matched 6485,6485: curl-clients-block
[6431@37896]ips_match_rule: matched rule 6485 6485 curl-clients-block (weight:40) <------"confirms curl custom signature is matching"
[6431@37896]ips_match_candidates: set best rule 6485 6485 curl-clients-block
[6431@37896]ips_set_pkt_verdict: action=DROP <------- "Best policy match correctly results in DROP"
[6431@37896]ips_set_pkt_verdict: turn tcp drop to DROP_SESSION <---------"Session drop action confirmed"
[6431@37896]ips_report_alert_va_internal: v_id=6485, a_id=6485, log=1, log_pkt=0
[6431@37896]ips_log: id=6485 conf=0x84, action=1
Based on the analysis of the debugs discussed above, if any of the specific debug sections indicate possible mismatch in the pattern/protocol/context, etc, tune the custom signature accordingly in an iterative approach, until the expected match and policy action is observed.
Related document:
|
