| Description |
This article explores common issues with VIPs configured on FortiGate.
In this troubleshooting guide, the real server IP is 192.168.4.5, and the masqueraded IP is 200.200.200.200. |
| Scope |
FortiGate v6.4, v7.0 and v7.2. |
| Solution |
In the following examples, the goal is to successfully access https://200.200.200.200:4444/.
Scenario 1: After VIP configuration, the application is not accessible in the browser from the IP.
Running Sniffer in this example discovered that traffic was one-way only and the app server was not responding to tcp:4444.
> diagnose sniffer packet any "host 200.200.200.200 && host 197.1.1.254 && port 4444" 4 0 l
2022-10-23 05:32:37.745752 port1 in 197.1.1.254.60688 -> 200.200.200.200.4444: syn 2485841736
2022-10-23 05:32:37.745785 port1 in 197.1.1.254.60690 -> 200.200.200.200.4444: syn 2814091578
Run a debug flow and check the session table (the output below is abridged):
> diag debug reset > diag debug flow filter saddr 197.1.1.254 > diag debug flow filter daddr 200.200.200.200 > diag debug flow filter dport 4444 > diag debug flow show function-name enable > diag debug flow trace start 20 > diag debug enable d=20085 trace_id=60 func=print_pkt_detail line=5810 msg="vd-root:0 received a packet(proto=6, 197.1.1.254:60706->200.200.200.200:4444) from port1. flag [S], seq 366366630, ack 0, win 64240"
id=20085 trace_id=60 func=resolve_ip_tuple_fast line=5891 msg="Find an existing session, id-00058163, original direction"
id=20085 trace_id=60 func=__ip_session_run_tuple line=3522 msg="DNAT 200.200.200.200:4444->192.168.4.5:4443"
Session list (the output below is abridged):
> diagnose sys session filter clear > diagnose sys session filter dport 4444 > diagnose sys session filter dst 200.200.200.200 > diagnose sys session filter src 197.1.1.254 > diagnose sys session list session info: proto=6 proto_state=02 duration=2 expire=8 timeout=3600 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/255
state=log may_dirty f00
statistic(bytes/packets/allow_err): org=120/2/1 reply=0/0/0 tuples=2
tx speed(Bps/kbps): 50/0 rx speed(Bps/kbps): 0/0
orgin->sink: org pre->post, reply pre->post dev=3->4/4->3 gwy=192.168.4.5/0.0.0.0 <<<< The source gateway is not populated
hook=pre dir=org act=dnat 197.1.1.254:60712->200.200.200.200:4444(192.168.4.5:4443)
hook=post dir=reply act=snat 192.168.4.5:4443->197.1.1.254:60712(200.200.200.200:4444)
pos/(before,after) 0/(0,0), 0/(0,0)
misc=0 policy_id=1 auth_info=0 chk_client_info=0 vd=0
Here, the application server was listening on port 4443 and not 4444. The following symptoms are expected even with a successful application connection:
1. It’s normal for a ping to the internal server IP from the internet to fail. In the below example, a ping to the real IP configured on the server failed, but a ping to the private IP still succeeded:
It's possible to ping 200.200.200.200 from the PC, even when the application is not reachable.
2. The targeted IP from the internet must be DNATed to the internal/private IP. This can be seen in “debug flow” logs:
3. The session list is expected to have the correct in/out interface with a correct DNAT/SNAT:
After correcting the port, https://200.200.200.200:4444/ becomes accessible:
Check the following if the expected result does not occur:
1. Policy Based Routing: PBR has precedence over the routing table and can cause the return route to exit through an undesired ISP (or interface). Mistakenly stating a wrong Gateway in PBR will also result in issues. Ensure PBR is not responsible. Session list logs will show the gateway for the return route.
2. Asymmetric routing issues: If the user has two ISPs, return traffic may follow a different ISP from the one traffic entered. This will be visible in a session list output and a debug flow.
3. In some environments, NAT is required on the VIP firewall policy (towards the server).
4. Ensure the Firewall policy is matched. Use “Policy lookup” tool on the FortiGate GUI:
5. Ensure the VIP object has a hit and that the hits increase as you try to access the server behind VIP repeatedly.
6. Ensure both DNAT and SNAT are configured correctly, as the server real IP is private and cannot be directly accessed from an internet domain. Ensure the domain is translated back to a public IP (see this information in a session list output, such as the one provided at the beginning of this article).
Commands/tools used in this article
1. Sniffer: diagnose sniffer packet any "host 200.200.200.200 && host 197.1.1.254 && port 4444" 4 0 l
2. Debug flow:
diag debug reset diag debug flow filter saddr 197.1.1.254 diag debug flow filter daddr 200.200.200.200 diag debug flow filter dport 4444 diag debug flow show function-name enable diag debug flow trace start 20 diag debug enable
3. Session list:
diagnose sys session filter clear diagnose sys session filter dport 4444 diagnose sys session filter dst 200.200.200.200 diagnose sys session filter src 197.1.1.254 diagnose sys session list
4. Policy lookup: See the FortiGate GUI.
5. Policy route:
diagnose ip proute match 200.200.200.200 197.1.1.254 port1 6 4444
Note: for the above command, use ? on Fortigate to see the next parameter to provide.
6. Routing table
get router info routing-table all The problem can likely be found in one of the above solutions. |
