Technical Tip: Carried Grade NAT (CGNAT) Dynamic IP Consistency

[Ahmed_M]

Description	This article describes an issue where ‘Dynamic IP consistency’ for Carrier Grade NAT (CGNAT) is not working as expected. In this scenario, CGNAT resource allocation IP Pools are configured with a range of external IP addresses, and the expectation is that a given client should consistently receive port-block-allocations (PBAs) for the same external IP. However, users may observe that they are getting PBAs from multiple public IPs within the IP Pool, which can cause problems.
Scope	FortiGate version 6.2.6, 7.0.5, 7.2.1 and later. NP7 Hyperscale Firewall.
Solution	Background: Hyperscale firewall CGNAT configuration starts by creating one or more CGN resource allocation IP pools. There are five different types or modes of CGNAT resource allocation IP pools, though only four support the 'Dynamic IP consistency' feature. Refer to this link for documentation regarding each mode and the supported features: https://docs.fortinet.com/document/fortigate/7.2.4/hyperscale-firewall-guide/544642/cgn-resource-all... In many deployments, service providers might restrict access to their services to a single source IP, so users may encounter outages and service issues if their sessions going to the same destination are source NAT’d with multiple external IPs. ‘Dynamic IP consistency’ fixes this issue by ensuring that a client is always allocated PBAs from the same public IP address, if possible. A definition of "Dynamic IP Consistency" is available here: https://docs.fortinet.com/document/fortigate/7.2.4/hyperscale-firewall-guide/903729/dynamic-ip-consi... Bear in mind that 'Dynamic IP consistency' is dependent on the 'hash-config' setting under ‘config system npu’, which is used to configure how the internal switch fabric (ISF) load-balances sessions across NP7 processors.   The default setting for ‘hash-config’ is dependent on the number of NP7 processors present on the FortiGate (‘5-tuple’ for models with even number of NP7 processors and ‘src-dst-ip’ for models with an odd number of NP7 processors). The following show the CLI commands that will display this information, as well as sample output from a FortiGate-3500F:  FortiGate-3500F (global) # dia npu np7 info  SN              : FG3K5Fxxxxxxxx  nr_chip         : 3      np_0          : 0000:2a:00.0    np_1          : 0000:69:00.0    np_2          : 0000:aa:00.0  FortiGate-3500F (global) # config  sys npu  FortiGate-3500F (npu) # get | grep hash  hash-config         : src-dst-ip    hash-tbl-spread     : enable  One issue that can occur when ‘hash-config’ is set to ‘5-tuple’ or ‘src-dst-ip' is that sessions from a given client may be distributed across multiple NP7 processors. While this is beneficial for performance, it can result in that one client’s sessions being NAT’ed across multiple public IPs.  To resolve this issue, it is suggested to set ‘hash-config’ to ‘src-ip’, which results in all sessions from a given source IP to be processed by the same NP7 processor:  # config system npu      set hash-config src-ip  end     ! BE AWARE THAT CHANGING ‘hash-config’ WILL CAUSE THE FORTIGATE TO RESTART!  Verification: The following example demonstrates a CGNAT configuration on a FortiGate-3500F and the impact of the ‘hash-config’ option: First, a CGNAT IP Pool set to ‘overload with port-block-allocation’ mode is created: # config firewall ippool     edit "Dynamic-Nat-pool"         set type cgn-resource-allocation         set startip 10.200.200.1         set endip 10.200.200.3         set arp-intf "Vlan901"         set cgn-overload enable     next end Next, the IP Pool is applied to a firewall policy: # config firewall policy     edit 1         set name "Test-policy"         set srcintf "Vlan-900"         set dstintf "Vlan901"         set action accept         set srcaddr "all"         set dstaddr "all"         set service "ALL"         set nat enable         set ippool enable         set poolname "Dynamic-Nat-pool"     next end In the following debug output, traffic was sent through the FortiGate-3500F from client 172.19.0.20 when ‘hash-config’ was set to the default of ‘src-dst-ip’. The output indicates that multiple PBAs were allocated across multiple external IPs: FortiGate-3500F (test-hw1) # dia fire ippool get-pub 172.19.0.20 Query private IP 172.19.0.20 np-0 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 5117-5244 np-0 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 5629-5756 np-0 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 6397-6524 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25213-25340 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25981-26108 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 26493-26620 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 45437-45564 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 46205-46332 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.2 port: 46845-46972 The above example indicates that the first three sessions were allocated to the NP7_0 processor and SNAT IP address of 10.200.200.2. The next three sessions were then offloaded to NP7_1 with SNAT IP 10.200.200.1, and finally, the last three sessions were offloaded to NP7_2 with SNAT  IP 10.200.200.2 Finally, the following debug output was taken after ‘hash-config’ was set to ‘src-ip’ and the FortiGate was restarted: FortiGate-3500F (test-hw1) # dia firewall ippool get-pub 172.19.0.20 Query private IP 172.19.0.20 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25213-25340 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25341-25468 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25469-25596 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25597-25724 np-1 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 25725-25852 FortiGate-3500F (test-hw1) # dia firewall ippool get-pub 172.19.0.19 Query private IP 172.19.0.19 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 45437-45564 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 45693-45820 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 45949-46076 np-2 policy-1 pool-8(PBA) public IP: 10.200.200.1 port: 46205-46332 As per the output, all sessions from 172.19.0.20 are being handled by NP7_1 and the NAT’ed IP address (10.200.200.1), same with all sessions from 172.19.0.19 are being handled by NP7_2 and NAT'ed IP address (10.200.200.1) remains consistent.