Regarding: Technical Note: Traceroute behaviour with NP2/NP4 ASIC based FortiGate devices

shin_sterneck · ‎10-21-2014

This is regarding: Technical Note: Traceroute behaviour with NP2/NP4 ASIC based FortiGate devices

Fortinet KB:

When performed the multiple traceroute from same source to destination, the FortiGate will show up as the first hop in the first traceroute, then it will start timing out when it did the traceroute again. This is expected using a Windows computer as it uses the ICMP protocol for the traceroute. In the first traceroute command output it can be seen that the FortiGate IP is shown as a hop, while in the later traceroute command the FortiGate IP is not seen as hop. The reason is that once the session is off-loaded onto the ASIC card, the ASIC card cannot generate the TTL exceeded msg back to source because ASIC cards are not programmed like that, so the client reports a time out. This is not an issue at all. If you clear the existing session or let the session expire after 1 minute (for ICMP), then again perform the traceroute, the FortiGate will appear as a hop, then the next traceroute will not show the FortiGate as a hop until the session is active/present. If you perform the traceroute using the Linux/Unix which uses the UDP port (destination port 33434-33464), every traceroute command will show the FortiGate as a hop, the reason is that every traceroute command will have different source and destination ports thus requiring the kernel to process the packets with ttl of 1. This feature can be tested by enabling the 'set auto-asic offload enable' command in the firewall policy then all traceroute attempts will show the FortiGate as a hop.

We recently came across this phenomenon and investigated why this is happening. I found the above technical note and am questioning the root cause of this and hence provided feedback directly at the KB note but as this feedback is not open I decided to share this here as well:

My Feedback:

I don't think that this is because windows uses ICMP and Linux uses UDP. Linux can produce ICMP trace/echo requests and responses are properly produced and returned from the Fortigate device every time, but its rather how ICMP packets are created and how the Fortigate device identifies different sessions.

A closer look at the differences between Windows and Linux, both issuing ICMP echo requests with their respective TTLs, show that the windows 7 system always specifies 0x0001 as the identifier for all sessions within the ICMP echo request packet (appears to be synchronized or set system wide), while Linux uses separate identifiers for each traceroute session/process within the ICMP packet.

I think that this is the reason why the Fortigate system (and maybe then indirectly the NP) is not able to distinguish between the different sessions.

Test Scenario: Host (10.10.40.43) --> Gateway (10.10.40.1) --> Fortigate (10.10.18.135) --> Destination Server (10.10.32.57)

Following two separate "ICMP" traces from windows and from Linux:

First trace from Windows host 10.10.40.43 to 10.10.32.57: ==========================================================

Hop 1 (3 requests):

1 0.000000 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=154/39424, ttl=1 2 0.000501 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit) 3 0.001561 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=155/39680, ttl=1 4 0.001861 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

5 0.003079 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=156/39936, ttl=1 6 0.003343 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

Hop 2 (3 requests):

7 1.009379 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=157/40192, ttl=2 8 1.009899 10.10.18.135 -> 10.10.40.43 ICMP 134 Time-to-live exceeded (Time to live exceeded in transit)

9 1.011283 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=158/40448, ttl=2 10 1.011889 10.10.18.135 -> 10.10.40.43 ICMP 134 Time-to-live exceeded (Time to live exceeded in transit)

11 1.013096 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=159/40704, ttl=2 12 1.013312 10.10.18.135 -> 10.10.40.43 ICMP 134 Time-to-live exceeded (Time to live exceeded in transit)

Hop 3 (3 requests):

13 2.023122 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=160/40960, ttl=3 14 2.023887 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=160/40960, ttl=62 (request in 13)

15 2.025417 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=161/41216, ttl=3 16 2.026121 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=161/41216, ttl=62 (request in 15)

17 2.027457 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=162/41472, ttl=3 18 2.027691 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=162/41472, ttl=62 (request in 17)

Second trace from Windows host 10.10.40.43 to 10.10.32.57: ==========================================================

Hop 1 (3 requests):

19 5.656561 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=163/41728, ttl=1 20 5.657117 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

21 5.658274 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=164/41984, ttl=1 22 5.658551 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

23 5.659556 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=165/42240, ttl=1 24 5.659823 10.10.40.1 -> 10.10.40.43 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

Hop 2 (3 requests):

25 6.670449 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=166/42496, ttl=2 26 10.430822 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=167/42752, ttl=2 27 14.433839 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=168/43008, ttl=2 ==> no response to all 3 icmp echo request packets, that should come from Fortigate.

Hop 3 (3 requests):

28 18.433602 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=169/43264, ttl=3 29 18.433836 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=169/43264, ttl=62 (request in 28)

30 18.435441 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=170/43520, ttl=3 31 18.435681 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=170/43520, ttl=62 (request in 30)

32 18.436866 10.10.40.43 -> 10.10.32.57 ICMP 106 Echo (ping) request id=0x0001, seq=171/43776, ttl=3 33 18.437159 10.10.32.57 -> 10.10.40.43 ICMP 106 Echo (ping) reply id=0x0001, seq=171/43776, ttl=62 (request in 32)

In the above two cases you see that the windows system assigns id=0x0001 to both trace sessions and no response on the 2nd "session" from the fortigate device.

Sessions on the Fortgate device: ================================

# get system session list | grep icmp
icmp 55 10.10.40.43:1 - 10.10.32.57:8 -

Now the same ICMP trace from Linux machines:

Test Scenario: Host (10.10.40.37) --> Gateway (10.10.40.37) --> Fortigate (10.10.18.135) --> Destination Server (10.10.32.57)

# Command used: "traceroute -I 10.10.32.57" (-I makes traceroute use ICMP instead of UDP)

First trace from Linux host 10.10.40.37 to 10.10.32.57: ==========================================================

Hop 1 (3 requests):

1 0.000000 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=0/0, ttl=1 2 0.000530 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit) 3 0.000659 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=1/256, ttl=1 4 0.000959 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit) 5 0.001044 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=2/512, ttl=1 6 0.001326 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

Hop 2 (3 requests): 7 0.001427 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=3/768, ttl=2 8 0.001983 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit) 9 0.002064 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=4/1024, ttl=2 10 0.002334 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit) 11 0.002405 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=5/1280, ttl=2 12 0.002948 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit)

Hop 3 (3 requests): 13 0.003036 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=6/1536, ttl=3 14 0.003683 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d03, seq=6/1536, ttl=62 (request in 13) 15 0.003759 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=7/1792, ttl=3 16 0.004411 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d03, seq=7/1792, ttl=62 (request in 15) 17 0.004482 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d03, seq=8/2048, ttl=3 18 0.004749 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d03, seq=8/2048, ttl=62 (request in 17)

Second trace from Linux host 10.10.40.37 to 10.10.32.57: ==========================================================

Hop 1 (3 requests):

19 1.640121 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=0/0, ttl=1 20 1.640630 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit) 21 1.640788 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=1/256, ttl=1 22 1.641072 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit) 23 1.641168 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=2/512, ttl=1 24 1.641442 10.10.40.1 -> 10.10.40.37 ICMP 70 Time-to-live exceeded (Time to live exceeded in transit)

Hop 2 (3 requests): 25 1.641558 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=3/768, ttl=2 26 1.642181 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit) 27 1.642251 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=4/1024, ttl=2 28 1.642520 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit) 29 1.642650 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=5/1280, ttl=2 30 1.643169 10.10.18.135 -> 10.10.40.37 ICMP 94 Time-to-live exceeded (Time to live exceeded in transit)

Hop 3 (3 requests): 31 1.643254 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=6/1536, ttl=3 32 1.643888 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d05, seq=6/1536, ttl=62 (request in 31) 33 1.643956 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=7/1792, ttl=3 34 1.644629 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d05, seq=7/1792, ttl=62 (request in 33) 35 1.644699 10.10.40.37 -> 10.10.32.57 ICMP 66 Echo (ping) request id=0x2d05, seq=8/2048, ttl=3 36 1.644962 10.10.32.57 -> 10.10.40.37 ICMP 66 Echo (ping) reply id=0x2d05, seq=8/2048, ttl=62 (request in 35) Sessions on the Fortgate device: ================================

icmp 14 10.10.40.37:11523 - 10.10.32.57:8 - 
icmp 16 10.10.40.37:11525 - 10.10.32.57:8 -

Here you can see two separate sessions with the corresponding ICMP IDs of "0x2d03 == 11523" and "0x2d05 == 11525" so the Fortigate can distinguish different ICMP "sessions" using this field and properly respond separately to each session just like with UDP traces (the RFC does seem to keep this ICMP ID field optional).

Other version of windows are known to use different, but fixed to the version, ICMP identifier values as well....

funny, no?

Shin

Rewanta_FTNT · ‎10-29-2014

Hi,

different ICMP identifier will have different session setup so when tracertoue pkts come reach fortigate, kernel can respond as the session will not offloaded to np2/np4. however in the case where session is offloaded when using windows(uses the same icmp identifier), built-in traceroute behavior are different for the repetitive traceroute tests. in case when udp/tcp used in traceroute (that they uses the random port) every traceroute pkts will have the different session setup so that each pkts are different, there by no session offload and can see the traceroute output when tracertoue are repeated. as you see, this is protocol behavior.

Thanks.

Rewan