tracert not display the correct hop

[sean3]

we have 4 fortigates run v7.0.14 build0601 (Mature), one per each WAN site.

I tracert from a host behind one fortigate of site A, to a target host behind the fortigate of site B

C:\Users\ADM-name>tracert 10.80.131.12

Tracing route to hsot.company.com [10.80.131.12]
over a maximum of 30 hops:

1 199 ms <1 ms 26 ms 10.250.14.1
2 <1 ms <1 ms <1 ms 172.29.7.22
3 216 ms 216 ms 216 ms 10.250.29.15 (Y)
4 243 ms 242 ms 242 ms 172.16.10.17
5 242 ms 242 ms 242 ms host.company.com [10.80.131.12]

the 3rd hop is not the desired one by SD-WAN. We have 2 tunnel-type interface (let's say X and Y) in SD-WAN outgoing interface bundle. The tracert shows Y but actually X is the selected outgoing interface.

and if we do diag sniffer with interface X, it has traffic result, and no traffic result if diag sniffer with interface Y, meaning traffic is going through the correct interface.

what is the logical of fortigate replying tracert request? Why it is like this?

[AEK]

Hi Sean

If I understand well I think your tracert query is going through the right path (X) but the tracert response is coming back from Y. You can confirm with sniffer.

[sean3]

thanks,

I did the diag sniffer tunnel-interface-name 'src host source-ip and dst host destination-ip' 4 0 a on both side, the traffic is going out and coming back via X, no traffic is going through Y.

[AEK]

Can you run this command from FG:

diagnose sniffer packet any 'host x.x.x.x and (icmp or udp)' 4

(where x.x.x.x is the IP of the host from which you run tracert, don't add destination host)

Then run tracert from that host and share the output. I think it will show more interesting information.

(if you see some irrelevant output like DNS queries/responses you can remove them from the output)

[sean3]

FYI. I have run the command on both FW of each Site, OL_MPLS_ZHA_212 and OL_MPLS_SKO_212 are the desired tunnel that we want for the traffic forwarding.

FW_Site_A $ diagnose sniffer packet any 'host 10.250.14.10 and (icmp or udp)' 4

Line 66: 24.807513 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 70: 24.808226 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 74: 24.808856 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 88: 25.813192 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 89: 25.813216 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 94: 26.028285 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 95: 26.028290 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 100: 26.243383 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 101: 26.243387 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 106: 27.256847 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 107: 27.256853 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 112: 27.500631 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 113: 27.500635 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 118: 27.743848 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 119: 27.743853 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 126: 28.759479 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 127: 28.759483 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 138: 29.001513 OL_MPLS_ZHA_212 in 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 139: 29.001525 VLAN600 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 140: 29.001526 UplinkToLAN out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 141: 29.001527 x1 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 142: 29.002748 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 143: 29.002760 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 144: 29.244878 OL_MPLS_ZHA_212 in 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 145: 29.244885 VLAN600 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 146: 29.244886 UplinkToLAN out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 147: 29.244887 x1 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 148: 29.246238 VLAN600 in 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 149: 29.246244 OL_MPLS_ZHA_212 out 10.250.14.10 -> 10.80.131.12: icmp: echo request
Line 150: 29.489148 OL_MPLS_ZHA_212 in 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 151: 29.489155 VLAN600 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 152: 29.489156 UplinkToLAN out 10.80.131.12 -> 10.250.14.10: icmp: echo reply
Line 153: 29.489157 x1 out 10.80.131.12 -> 10.250.14.10: icmp: echo reply

===

FW_Site_B $ diagnose sniffer packet any 'host 10.80.131.12 and (icmp or udp)' 4

Line 1: zjk-fw-01 $ diagnose sniffer packet any 'host 10.80.131.12 and (icmp or udp)' 4
Line 3: filters=[host 10.80.131.12 and (icmp or udp)]
Line 4: 50.201872 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 5: 50.201967 VLAN600 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 6: 50.201971 UplinkToLAN out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 7: 50.201976 x2 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 8: 50.202900 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 9: 50.202942 VLAN600 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 10: 50.202945 UplinkToLAN out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 11: 50.202948 x2 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 12: 50.203714 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 13: 50.203756 VLAN600 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 14: 50.203759 UplinkToLAN out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 15: 50.203762 x2 out 172.16.10.30 -> 10.80.131.12: icmp: time exceeded in-transit
Line 16: 51.238945 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 17: 51.239870 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 18: 51.481661 OL_MPLS_SKO_212 in 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 19: 51.481692 VLAN600 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 20: 51.481696 UplinkToLAN out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 21: 51.481700 x2 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 22: 51.483027 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 23: 51.483149 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 24: 51.724892 OL_MPLS_SKO_212 in 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 25: 51.724910 VLAN600 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 26: 51.724913 UplinkToLAN out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 27: 51.724917 x2 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 28: 51.726262 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 29: 51.726354 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 30: 51.968015 OL_MPLS_SKO_212 in 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 31: 51.968034 VLAN600 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 32: 51.968037 UplinkToLAN out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 33: 51.968040 x2 out 10.250.29.20 -> 10.80.131.12: icmp: time exceeded in-transit
Line 34: 52.786229 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 35: 52.786332 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 36: 53.028295 OL_MPLS_SKO_212 in 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 37: 53.028325 VLAN600 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 38: 53.028330 UplinkToLAN out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 39: 53.028335 x1 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 40: 53.029690 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 41: 53.029826 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 42: 53.271835 OL_MPLS_SKO_212 in 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 43: 53.271854 VLAN600 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 44: 53.271857 UplinkToLAN out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 45: 53.271861 x1 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 46: 53.273227 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 47: 53.273317 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 48: 53.515051 OL_MPLS_SKO_212 in 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 49: 53.515073 VLAN600 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 50: 53.515077 UplinkToLAN out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 51: 53.515080 x1 out 172.29.7.17 -> 10.80.131.12: icmp: time exceeded in-transit
Line 52: 54.339101 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 53: 54.339237 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 54: 54.581120 OL_MPLS_SKO_212 in 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 55: 54.581221 VLAN600 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 56: 54.581227 UplinkToLAN out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 57: 54.581231 x2 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 58: 54.582477 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 59: 54.582546 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 60: 54.824445 OL_MPLS_SKO_212 in 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 61: 54.824548 VLAN600 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 62: 54.824554 UplinkToLAN out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 63: 54.824558 x2 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 64: 54.825773 VLAN600 in 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 65: 54.825842 OL_MPLS_SKO_212 out 10.80.131.12 -> 10.250.14.10: icmp: echo request
Line 66: 55.067709 OL_MPLS_SKO_212 in 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 67: 55.067926 VLAN600 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 68: 55.067933 UplinkToLAN out 10.250.14.10 -> 10.80.131.12: icmp: echo reply
Line 69: 55.067938 x2 out 10.250.14.10 -> 10.80.131.12: icmp: echo reply

[dbhavsar]

Hello @sean3 ,

- Could you please confirm the tunnels you created are in the SDWAN zone?

[sean3]

yes, it is confirmed.

One thing we found is that Administrative Access PING is not ticked under the tunnel-type interface, which means this interface might not be allowed to access by PING(tracert including). But I am sure there must be something else to consider, since other sites with PING enabled is still showing an undesired hop in the path, but diag sniffer are all showing the desired interface/hop in the path.

[Toshi_Esumi]

Is this what you're describing? And, 10.250.29.15 is configured as tunnel Y's interface IP, right?

Toshi

[sean3]

you are really awesome.

yes exactly.

from SD-WAN it should display tunnel X' IP as one of the tracert hop, but it shows 10.250.29.15 as one of the hop.

both tunnel X and tunnel Y are added into the sd-wan outgoing interface bundle already. and diag sniffer shows the traffic is flowing through tunnel X.

[Toshi_Esumi]

I set up below test environment with two FGTs(70F, 60E-POE) v7.2.8. Then set up an SD-WAN rule to route traffic destined to 172.20.1.0/24 to tunnel-x only. But the traceroute was below as expected. So I couldn't recreate your problem.

C:\Users\email>tracert 172.20.1.11

Tracing route to TOSHIWIN10LP [172.20.1.11]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.99
2 1 ms 1 ms 1 ms 10.100.0.2
3 2 ms 1 ms 1 ms TOSHIWIN10LP [172.20.1.11]

I feel your environment is not the same as I drew.

Toshi