Traceroute via FGT unit doesn't show as a hop

[Aigarz]

Hello,

I've come across fgt100d HA

 installation where internal Ip doesn't show in traceroute

What I get.

>tracert -d 8.8.8.8   1     1 ms     2 ms     1 ms  2.2.2.2   2     4 ms     3 ms     3 ms  1.1.1.1

... ^C

What I would like to get

>tracert -d 8.8.8.8   1     1 ms     1 ms     1 ms  192.168.100.1   2     1 ms     2 ms     1 ms  2.2.2.2   3     4 ms     3 ms     3 ms  1.1.1.1 ^C

Setup is quite basic.

there are several Lan interfaces combined under one zone. Policy is

LANzone > wan all traffic is permitted

192.168.100.1 responds to icmp

I'm suspecting fortigate doesn't decrease TTL or it's somehow connected that traffic is processed using NP...  Any ideas what could be switched on/off ;) ?

[emnoc]

1st

is the traceroute udp or icmp based

2nd

The diag debug flow  is your best friend

3rd

what's your allowaccess statement on the in-comming interface for the traceroute?

here's a tracerouet unix with udp and then icmp, 10.52.1.1 is my edge-fgt btw, you will see that reported in the ICMP based traceroute

[Aigarz]

Thanks for looking into this. 1st trace issued from windows, should be ICMP 3rd while testing: set allowaccess ping https snmp http fgfm As from your trace with UDP it's obviously blocked, therefore showing  * * * In my scenario - hop simply doesn't show up. If I would "translate" it to you screenshot - it would be ... 4. 10.255.255.6 5. 10.255.255.5 ----------10.52.1.1----- missing 6. <public network> nb: I went over similar installations also 100d  - first hop is showing properly - direct gateway shows as a first hop(Fortigate internal IP). Only differences I can spot  in this case - there are several "Vlan Interfaces" bundled into one "Zone" and then security policies are made from "Zone" to WAN1 Actually this is the first time I've came across this behaviour on Fortigate. I recall over years there were boxes (non Fortigate) where you could artificially manipulate TTL settings to  make L3 Firewalls "dissapear" from traceroutes. But in this case customer didn't touch any special knobs or settings. (hopefully :))

[emnoc]

What did diag debug flow show? I was also going to suggest  a NPU issue but a 100D pretty much does not have any NPU.

FWIW a FGT140D ( similar ) that's running   5.2.4 in my lab works great with udp so I do believe you are on to something with the 1st hop being dropped by the FGT.

e.g

LAB# traceroute 8.8.8.8 port 666  <I tried my own port # for the heck of it traceroute to 8.8.8.8 (8.8.8.8), 30 hops max, 40 byte packets  1  10.64.1.1 (10.64.1.1)  0.645 ms  0.943 ms  2.359 ms  2  10.64.18.30 (10.64.18.30)  2.388 ms  0.983 ms  2.294 ms  <-FGT  3  1.1.1.1 (1.1.1.1)  2.462 ms  3.677 ms  2.552 ms   <- ISP  4  * * *  5  * * *  6  24.175.43.223 (24.175.43.223)  26.671 ms  17.369 ms  17.125 ms  7  24.175.41.46 (24.175.41.46)  20.408 ms  26.485 ms 24.175.41.50 (24.175.41.50)  25.853 ms  8  66.109.6.88 (66.109.6.88)  20.213 ms  22.052 ms  23.949 ms  9  107.14.17.236 (107.14.17.236)  18.466 ms  19.814 ms 107.14.17.232 (107.14.17.232)  23.181 ms 10  66.110.57.97 (66.110.57.97)  41.224 ms  36.653 ms  31.729 ms 11  * *c * 12  *

The TTL is set by the client so I don't see how a FGT or any device will adjust the TTL  btw.

[awasfi_FTNT]

Hello,

I think you can see the FortiGate as a hop on your traceroute if you disable ASIC offloading on the firewall policy.

It can be done from CLI only:

config firewall policy

edit <policy_ID>

set auto-asic-offload disable

next

end

Or instead of disabling it just wait one minute for the session to be cleared then run traceroute again, you should see the FortiGate as a hop on first traceroute.

Note that it's not recommended to disable it permanently. Just temporary for testing, once test done re-enable it as this will affect the device performance.

ASIC can't generate the TTL exceeded msg back to source, so client report timed out.  While on Linux UDP traceroute it will shows Fortigate as hop, since each command has a different source and destination ports which will be processed by the kernel.

Regards,

[emnoc]

I think you can see the FortiGate as a hop on your traceroute if you disable ASIC offloading on the firewall policy.

Never had to disable that on a policy. In fact I don't think  traceroute  offloaded to begin with with ICMP if that was the case every  FGT firewall would require what your suggesting.

Either way you can monitor the session table  the  offload flags from the cli.

Ken

[supafin]

Bumping old topic.

After upgrading our 100D from 5.2.5 to current 5.2.11 we have this behavior that when doing trace thru FGT, first packet gets correct reply from FGT but after that, it doesn't respond anymore and hop disappears from trace.

It sounds like first packet is handled by cpu and rest by Asic.

However, I can't find "auto-asic-offload" option anywhere under firewall policy.

When doing 'diag sys session list' there is no NPU tag or similar that would clearly indicate icmp traffic being offloaded.

It seems this 100D model has CP8 Asic.

Is this issue that icmp traffic is offloaded and how to fix it so that it would be handled by cpu?

Maybe some global config?

Clearly there has been some changes since traceroute didn't behave like this with old firmware.

Thanks!

[emnoc]

Again.

A windows tracert using ICMP will show the fortigate as 1st hop, a unix traceroute using UDP will never show the fortigate as 1st hop  depending on what the policy is set for and how many hops away from  the src 

Regarding ICMP the  fortigate still needs to allow all ICMP or the correct ICMP for this all to happen.

Here's  a few   demos  ( FGT v5.2.11 )

ICMP  traceroute ( the same thing windows OSes does by default )

macbook:~ kfelix$ traceroute -I 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 72 byte packets  1  10.11.0.1 (10.11.0.1)  29.633 ms  10.269 ms  4.713 ms  2  10.255.254.22 (10.255.254.22)  1.672 ms  2.913 ms  1.566 ms  3  10.255.255.22 (10.255.255.22)  208.203 ms  201.552 ms  190.007 ms  4  10.255.255.6 (10.255.255.6)  203.844 ms  137.281 ms  79.122 ms  5  10.255.255.5 (10.255.255.5)  68.175 ms  77.159 ms  55.985 ms  6  10.52.1.1 (10.52.1.1)  51.844 ms  18.390 ms  21.070 ms   <----FORTIGATE  7  2xxxxxxxxxxxxxx)  26.835 ms  11.950 ms  11.791 ms  8  xxxxxxx.aus3.us.above.net (208.184.x.x)  21.789 ms  16.801 ms  10.877 ms  9  ae3.mpr1.aus1.us.zip.zayo.com (64.125.31.24)  12.339 ms  10.722 ms  8.309 ms 10  ae1.cr1.dfw2.us.zip.zayo.com (64.125.27.29)  22.519 ms  13.803 ms  15.036 ms 11  ae11.er1.dfw2.us.zip.zayo.com (64.125.20.66)  15.649 ms  24.570 ms  22.355 ms 12  72.14.194.53 (72.14.194.53)  14.759 ms  13.649 ms  16.286 ms 13  108.170.252.129 (108.170.252.129)  14.626 ms  17.175 ms  22.475 ms 14  108.170.230.241 (108.170.230.241)  13.945 ms  15.908 ms  13.938 ms 15  google-public-dns-a.google.com (8.8.8.8)  14.039 ms  13.484 ms  13.659 ms

NOW udp.prt 53 to google dns which is allowed by my firewall policy, ( the 1st  packet is is port 53 and incremented up by +1 )

macbook:~ kfelix$ traceroute  -p 53 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets  1  10.11.0.1 (10.11.0.1)  5.633 ms  19.344 ms  2.172 ms  2  10.255.254.22 (10.255.254.22)  1.961 ms  2.175 ms  1.389 ms  3  10.255.255.22 (10.255.255.22)  424.345 ms  427.892 ms  396.251 ms  4  10.255.255.6 (10.255.255.6)  407.848 ms  433.819 ms  523.280 ms  5  10.255.255.5 (10.255.255.5)  401.046 ms *

 6  * * *  7  * * *  8  * *

Now if you set a fwpoilicy for ALL  services ports in the firewall the behavior would be different;

e.g ( with all services )

macbook:~ kfelix$ traceroute  -p 53 8.8.8.8 traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 52 byte packets  1  10.11.0.1 (10.11.0.1)  7.658 ms  9.107 ms  2.258 ms  2  10.255.254.22 (10.255.254.22)  1.555 ms  3.262 ms  1.589 ms  3  10.255.255.22 (10.255.255.22)  447.564 ms  399.062 ms  417.110 ms  4  10.255.255.6 (10.255.255.6)  409.758 ms  513.274 ms  397.748 ms  5  10.255.255.5 (10.255.255.5)  182.166 ms * *  6  10.52.1.1 (10.52.1.1)  627.076 ms  428.114 ms  332.773 ms <---THE FGT AGAIN  7  xxxxxxxxxxxxxx  277.824 ms  249.232 ms  251.347 ms  8  xxxxxx.aus3.us.above.net (208.184.xxxx.xxxx)  253.721 ms  258.077 ms  287.120 ms  9  ae3.mpr1.aus1.us.zip.zayo.com (64.125.31.24)  302.571 ms  415.828 ms  366.882 ms 10  ae1.cr1.dfw2.us.zip.zayo.com (64.125.27.29)  357.037 ms  296.428 ms  292.251 ms 11  ae11.er1.dfw2.us.zip.zayo.com (64.125.20.66)  289.884 ms  338.082 ms  303.955 ms 12  * * *

This is pure  session-state-tracking and what a firewall does.

Ken

[supafin]

Yes I'm aware that FGT doesn't reply to UDP packet unless it's allowed. I'm using Windows with ICMP.

But this wasn't my point.

By "first packet" I didn't mean first hop but rather first traceroute. When doing same traceroute second time, hop has disappeared.

But I just realized that since we use this 100D as vpn endpoint, all destinations are reached thru ipsec tunnels.

And since ipsec is offloaded by CP8 Asic, could that explain behavior I was trying to explain?

All ipsec crypto devices in use: CP8:         null:   0       0         des:    0       0         3des:   26092729        19952511         aes:    0       0         aria:   0       0         seed:   0       0         null:   0       0         md5:    0       0         sha1:   26092767        19952511         sha256: 0       0         sha384: 0       0         sha512: 0       0 SOFTWARE:         null:   0       0         des:    0       0         3des:   0       0         aes:    0       0         aria:   0       0         seed:   0       0         null:   0       0         md5:    0       0         sha1:   0       0         sha256: 0       0         sha384: 0       0         sha512: 0       0

[emnoc]

Could be try disabling the offload  ( temporary )  but I don't think that's the issues.

Here's why;

Each traceoute is a new session ( run a diag debug flow trace and capture the :new session-: value } or  use diag sys session and a filter than list out the session and look at the serial number.

So I don't see how one  traceroute thats hop  works ad the other  does not, if we think it's a offloading issues. It could be something else going on. It's either offloaded or not but give it a try and the same behavior for that traffic flow.

Can you provide a show full of the policy that your hitting

e.g

show full firewall policy 777