Technical Tip: Troubleshooting port forwarding/Virtual IPs when using NGFW policy-based mode

[Dongfang_Li_FTNT]

Description	This article discusses troubleshooting techniques and common issues that can occur when configuring port forwarding/Virtual IPs (VIPs) in NGFW policy-based mode, including what to check for in terms of packet captures and debug flow trace output.
Scope	FortiGate; NGFW policy-based mode; Virtual IPs
Solution	As a primer on NGFW policy-based mode and configuring port forwarding/VIPs, refer to the following KB article: Technical Tip: How to configure port forwarding/Virtual IPs when using NGFW policy-based mode As discussed in the above article, NGFW policy-based mode enables Central SNAT automatically, which means that incoming connections destined for the VIP external IP address will automatically be translated to the real mapped IP address; there is no need to place the VIP object itself into any FortiGate policies. However, it is critical during VIP setup that administrators have matching policies in both the SSL Inspection & Authentication Policy section as well as the Security Policy section. If policies are not added to each section to accept VIP traffic based on the real mapped IP address as the destination, then the traffic will not be allowed through. NGFW policy-mode splits policy assessment into two stages that are handled by two separate modules, and this can make troubleshooting more challenging compared to NGFW profile-based mode: The FortiGate kernel handles policy matching for the SSL Inspection & Authentication Policies (defined under config firewall policy), and this policy assessment can be monitored using the FortiGate debug flow tools. This is the same as NGFW profile-based mode. However, NGFW policy-based mode handles Security Policy assessment using the IPS Engine, specifically a module called the Policy Match Engine (PME). Administrators who are used to troubleshooting NGFW profile-based mode may have some initial difficulty when troubleshooting policy matching in NGFW policy-based mode, as the debug flow will only ever show traffic as matching to one of the SSL Inspection & Authentication Policies. This can be even more confusing when considering that many NGFW policy-mode deployments will only ever utilize the single 'Default' SSL Inspection & Authentication policy that allows Any/Any traffic, so the debug flow will often say that many different traffic flows are all matching the same policy ID. Note that it is possible to debug the PME to see which Security Policy is matched, though in some cases it may be faster to perform a manual review of the configured Security Policy list due to the complex and dynamic nature of how the PME matches traffic to a given policy. This article will not discuss PME debug output due to its complexity, so for more guidance on debugging Security Policy assessment by the PME, refer to the following KB article instead: Technical Tip: How to check NGFW policy matching Troubleshooting Example: Consider the following example topology, where the FortiGate has a VIP with an external address of 100.64.0.1 and a mapped address of 192.168.189.2 (which corresponds to an on-site server). Note that in this case, there is no inbound Source NAT enabled (and therefore no Central SNAT policies to consider): Scenario 1: There is an SSL Inspection & Authentication policy, but no Security Policy. Running a packet capture in the CLI (diagnose sniffer packet [...]) shows that the FortiGate is receiving an ICMP ping from Client 100.64.0.100 on the wan1 interface, but it is not being forwarded out of the internal port1 interface: diagnose sniffer packet any 'host 100.64.0.100 and icmp' 4 interfaces=[any] filters=[host 100.64.0.100 and icmp] 12.311483 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 17.310280 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 22.314867 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 27.310445 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request      2. A debug flow trace is started on the FortiGate while the client's pings are coming in. The output indicates that traffic should be matched to a policy and sent to IPS: FortiGate # diagnose debug flow filter saddr 100.64.0.100 FortiGate # diagnose debug flow filter proto 1 FortiGate # diagnose debug flow trace start 100 FortiGate # diagnose debug enable id=20085 trace_id=13 func=print_pkt_detail line=5822 msg="vd-root:0 received a packet(proto=1, 100.64.0.100:1->100.64.0.1:2048) from wan1. type=8, code=0, id=1, seq=123." id=20085 trace_id=13 func=init_ip_session_common line=5993 msg="allocate a new session-00000eb2" id=20085 trace_id=13 func=fw_pre_route_handler line=181 msg="VIP-192.168.189.2:1, outdev-wan1" id=20085 trace_id=13 func=__ip_session_run_tuple line=3531 msg="DNAT 100.64.0.1:8->192.168.189.2:1" id=20085 trace_id=13 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-192.168.189.2 via port1" id=20085 trace_id=13 func=fw_snat_check line=513 msg="NAT disabled by central SNAT policy!" id=20085 trace_id=13 func=fw_forward_handler line=811 msg="Allowed by Policy-2:" id=20085 trace_id=13 func=ids_receive line=298 msg="send to ips" Solution:  In this case, traffic is matching the first stage of policy assessment (SSL Inspection & Authentication Policy #2 is matched and allowing traffic), and traffic is being sent to the IPS Engine for Security Policy matching, but it is NOT being forwarded out via port1 as expected. In this case, check the Security Policy list and see if an appropriate policy exists that allows traffic destined for the VIP's mapped IP address (192.168.189.2 in this scenario). If an appropriate match is not present, then add a policy and re-test. Scenario 2: There is a Security Policy, but no SSL Inspection & Authentication policy. Similar to Scenario 1, the FortiGate packet capture shows the traffic arriving on wan1 but not being forwarded out port1: diagnose sniffer packet any 'host 100.64.0.100 and icmp' 4 interfaces=[any] filters=[host 100.64.0.100 and icmp] 38.652590 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 43.273197 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 48.272051 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 53.283476 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request      2. Meanwhile in the debug flow trace, traffic is being dropped at this stage due to forward policy check failure: FortiGate # diagnose debug flow filter saddr 100.64.0.100 FortiGate # diagnose debug flow filter proto 1 FortiGate # diagnose debug flow trace start 100 FortiGate # diagnose debug enable id=20085 trace_id=17 func=print_pkt_detail line=5822 msg="vd-root:0 received a packet(proto=1, 100.64.0.100:1->100.64.0.1:2048) from wan1. type=8, code=0, id=1, seq=127." id=20085 trace_id=17 func=init_ip_session_common line=5993 msg="allocate a new session-00000f74" id=20085 trace_id=17 func=fw_pre_route_handler line=181 msg="VIP-192.168.189.2:1, outdev-wan1" id=20085 trace_id=17 func=__ip_session_run_tuple line=3531 msg="DNAT 100.64.0.1:8->192.168.189.2:1" id=20085 trace_id=17 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-192.168.189.2 via port1" id=20085 trace_id=17 func=fw_forward_handler line=655 msg="Denied by forward policy check (policy 0)" Solution: This scenario may be more familiar to administrators who frequently troubleshoot NGFW profile-based mode FortiGates, as the forward policy check message indicates that traffic did not match a created SSL Inspection & Authentication Policy (instead, it matched to the implicit-deny policy). To solve this, check and add an appropriate SSL Inspection & Authentication Policy. Scenario 3: There is both a Security Policy and an SSL Inspection & Authentication Policy available to match this traffic (i.e., working traffic flow). In the following packet capture, traffic is now successfully arriving on wan1 and is being forwarded out via port1: diagnose sniffer packet any 'host 100.64.0.100 and icmp' 4 interfaces=[any] filters=[host 100.64.0.100 and icmp] 26.754708 wan1 in 100.64.0.100 -> 100.64.0.1: icmp: echo request 26.754906 port1 out 100.64.0.100 -> 192.168.189.2: icmp: echo request 26.755280 port1 in 192.168.189.2 -> 100.64.0.100: icmp: echo reply 26.755478 wan1 out 100.64.0.1 -> 100.64.0.100: icmp: echo reply Meanwhile, the debug flow trace output shows that traffic is matching SSL Inspection & Authentication Policy #2, then being sent to IPS, and finally being sent out via port1. Note how the matched Security Policy ID never shows in the debug flow output (since it is handled solely by the IPS Engine's PME module). FortiGate # diagnose debug flow filter saddr 100.64.0.100 FortiGate # diagnose debug flow filter proto 1 FortiGate # diagnose debug flow trace start 100 FortiGate # diagnose debug enable id=20085 trace_id=21 func=print_pkt_detail line=5822 msg="vd-root:0 received a packet(proto=1, 100.64.0.100:1->100.64.0.1:2048) from wan1. type=8, code=0, id=1, seq=131." id=20085 trace_id=21 func=init_ip_session_common line=5993 msg="allocate a new session-00001025" id=20085 trace_id=21 func=fw_pre_route_handler line=181 msg="VIP-192.168.189.2:1, outdev-wan1" id=20085 trace_id=21 func=__ip_session_run_tuple line=3531 msg="DNAT 100.64.0.1:8->192.168.189.2:1" id=20085 trace_id=21 func=vf_ip_route_input_common line=2615 msg="find a route: flag=00000000 gw-192.168.189.2 via port1" id=20085 trace_id=21 func=fw_snat_check line=513 msg="NAT disabled by central SNAT policy!" id=20085 trace_id=21 func=fw_forward_handler line=811 msg="Allowed by Policy-2:" id=20085 trace_id=21 func=ids_receive line=298 msg="send to ips" id=20085 trace_id=21 func=ipd_post_route_handler line=490 msg="out port1 vwl_zone_id 0, state2 0x4000, quality 0. Related articles: Technical Tip: How to configure port forwarding/Virtual IPs when using NGFW policy-based mode Technical Tip: How to check NGFW policy matching FortiGate Parallal Path Processing - Packet flow ingress and egress