| Solution |
When the FortiGate is operating in NGFW policy-mode, traffic must first be matched to an SSL Inspection & Authentication Policy, then it must be matched to an appropriate Security Policy to determine what security inspection will be conducted. Notably, Security Policy assessment is handled by the Policy Match Engine (PME).
As part of the Security Policy matching process, it is possible to filter based on the Application signature that is matched to the traffic, as well as the FortiGuard Category Rating (aka URL category) associated with the destination (assuming it is web traffic):
The following is a summary of key PME behaviors to keep in mind before discussing the known-behavior further below:
- The type of scanning done by PME depends on the configuration of all Security Policies that can potentially be matched to by a given traffic flow.
- For example, if traffic can potentially match to a policy with URL Category filtering then the PME will perform a FortiGuard category rating lookup.
- Likewise, having a policy with Application filtering results in the PME scanning the traffic to match it to an Application Control signature.
- If these filters are not present in the list of potential policy matches then they are not performed by the PME (i.e. no URL Categories being set means that no FortiGuard rating lookup occurs).
- Each scanning type (category rating, signature matching, etc.) occurs in parallel, but the time needed to finish the scanning can vary.
- The scanning process is not instantaneous, as traffic must flow through the PME for a brief period so that it can assess the traffic and match the traffic to a Security Policy (or if there is no match, block the traffic).
- The PME will create a list of all Security Policies that can potentially match to a given traffic stream, but policy matching is prioritized in a top-down order (i.e. if two equally-matching policies are available then the entry higher on the list will be chosen).
For more in-depth info on PME and Security Policy matching, refer to the following Knowledge Base articles:
The Issue:
Two of the key behaviors mentioned above (time variances in scanning and top-down policy priority) can combine to create some unexpected behaviors. In lab testing of NGFW policy-mode, the following has been found to occur:
- In this scenario, assume that there are two Security Policies. One is filtered using Application signatures, the other is filtered based on URL Category. Assume that traffic flowing through the FortiGate could match to either policy.
- If the Application-filtered policy is placed above the URL Category then it can be possible for the traffic to be signature-matched to the Application policy before a FortiGuard Category rating is received (i.e. Application scanning often completes faster than FortiGuard Category ratings).
- Because the Application policy is above the URL Category policy, the PME will immediately match traffic to the Application Policy and will not wait for the FortiGuard Category Rating to complete.
- On the other hand, if the URL Category policy is placed above the Application policy then the FortiGate will actually wait until the FortiGuard category rating is received/completed before it makes a final assessment (even if an Application signature has already been matched to the traffic).
- Since the URL Category policy is placed above the Application policy, it will be the preferred match.
The first example is particularly notable because it can sometimes result in the FortiGate generating false-positive block logs. As noted in the Description field, these false-positive logs will indicate that traffic is being blocked due to FortiGuard category rating errors (msg='A rating error occurs' and 'error='unknown' in Web Filter logs) when in-fact traffic is flowing without issue. When checking PME debugs, it is possible to observe messages similar to the following:
2025-01-27 15:49:54 PME[86/1] matches: {
01 : 15895 60096 SSL
}
2025-01-27 15:49:54 PME[86/1] filtering policy 7 'Allow - App Sig'
2025-01-27 15:49:54 PME[86/1] ...explicit match 15895 SSL <------- App signature has been matched...
2025-01-27 15:49:54 PME[86/1] ...got explicit match 15895 60096 SSL: keep
2025-01-27 15:49:54 PME[86/1] ...keep
2025-01-27 15:49:54 PME[86/1] filtering policy 6 'Allow - Web Rating'
2025-01-27 15:49:54 PME[86/1] ...url unknown/awaiting <------ ...but web rating is not yet complete
2025-01-27 15:49:54 PME[86/1] ...keep
2025-01-27 15:49:54 PME[86/1] filtering policy 3 'Block URL'
2025-01-27 15:49:54 PME[86/1] ...url unknown/awaiting
2025-01-27 15:49:54 PME[86/1] ...keep
2025-01-27 15:49:54 PME[86/1] filtering policy 2 'Catch All'
2025-01-27 15:49:54 PME[86/1] ...explicit match 15895 SSL
2025-01-27 15:49:54 PME[86/1] ...got explicit match 15895 60096 SSL: keep
2025-01-27 15:49:54 PME[86/1] ...keep
2025-01-27 15:49:54 PME[86/1] policies 4 {
01 : 07 Allow - App Sig (explicit match) <------ Application Security Policy on top of list
02 : 06 Allow - Web Rating <------ URL Category Security Policy
03 : 03 Block URL
04 : 02 Catch All (explicit match)
}
2025-01-27 15:49:54 PME[86/1] match: app=15895 60096 SSL url=-1 UNKNOWN
2025-01-27 15:49:54 PME[86/1] matching policy 'Allow - App Sig'
2025-01-27 15:49:54 PME[86/1] ...matching apps
2025-01-27 15:49:54 PME[86/1] ...explicit match
2025-01-27 15:49:54 PME[86/1] ...matching actions
2025-01-27 15:49:54 PME[86/1] [EXPLICIT PASS] Allow - App Sig : 15895 60096 SSL url=-1
2025-01-27 15:49:54 PME[86/1] ...trigger policy 7 Allow - App Sig
2025-01-27 15:49:54 PME[86/1] [DECISION MADE] PASS view=3 policy=7 features={p:0x1005 s:0} <----- Note how Policy 7 'Allow - App Sig' is chosen despite the web rating request being incomplete.
2025-01-27 15:49:54 PME[86/1] policy=7 action=0 log_traffic=1 isdb_src/dst=0/0
2025-01-27 15:49:54 PME[86/1] policy=7 action=0 log_traffic=1 isdb_src/dst=0/0
Recommendations/Workarounds:
If traffic can potentially match both types of polices, it is recommended to ensure that a URL Category policy is placed at the top of the assessment list. Placing a URL Category at the top of the list ensures that the PME will wait to complete the FortiGuard category rating request before making a final decision on Security Policy matching (even if an Application signature has already been matched earlier in the process).
Additionally, it is crucial to note that the behavior mentioned above is controlled by the top-most Security Policy in the list of potential matches, so policies can be in any order as long as at least one with URL Categories is on top. For example, the following are examples of good configurations:
- Scenario 1 - Good:
- LAN->WAN - URL Category Policy
- LAN->WAN - Application Policy
- LAN->WAN - URL Category Policy
- Scenario 2 - Good:
- LAN->WAN - URL Category + Application Policy
- LAN->WAN - Application Policy
- LAN->WAN - URL Category Policy
- Scenario 3 - Potentially Problematic:
- LAN->WAN - Application Policy
- LAN->WAN - URL Category Policy
- Scenario 4 - Potentially Problematic:
- DMZ->WAN - URL Category Policy (note the different source interface, which removes it from the potential match list even though it is at the top).
- LAN->WAN - Application Policy
- LAN->WAN - URL Category Policy
Additional Note:
In cases where this issue has been reported, not all traffic will generate false-positive UTM blocks. In previous cases, the issue has been repeatable with Microsoft-related traffic (e.g. Microsoft. Portal, Microsoft.Teams, etc.), but it is unclear why this was the case. No other traffic seemed to trigger the false-positive logs, and again, no traffic seemed to actually be dropped.
Even when a local URL-based category is used, the firewall will still access the FortiGuard servers. This is by design.
|
