Description
This article describes specific behavior on selecting routes.
The result can then differ between offloaded and non-offloaded sessions and cause undesired behavior.
Solution
Whenever the kernel's routing table is updated, sessions using the route change will be triggered to evaluate the routing result again.
If the FortiGate is running a FortiOS version older than 7.0.16 GA, 7.2.8 GA and 7.4.4 GA all session will be re-evaluated due to the bug 970179, is explained here: Resolved Issues FortiOS 7.2.8.
This is to simply re-evaluate the routing based on current/updated information.
However, there is a behavior difference in NP offloaded sessions and non-offloaded session when making the decisions.
For an offloaded session, FortiGate will, in this order:
- Check if the route is still valid on the known output interface i.e. that there is still a valid route to that destination with the original output interface and the gateway has not changed. This session will not be marked dirty. Basically, no change is made.
- If the route does not exist or has changed, the session will be marked as dirty and the known original output interface will be dropped on the session. This will also clear the session from NPU (if offloaded).
The next packet will be checked against the routing table again to find the best match.
For non-offloaded sessions, FortiGate will skip step 1 (i.e. re-evaluating and keeping the session) and will just execute step 2, dropping the interface info and re-evaluating the first packet after the routing change.
For the non-offloaded session, if the routing updates affect live traffic, the result will be that traffic is suddenly routed differently.
This can affect the traffic in an undesirable way if the new, undesired routing information leads to a dead end for the particular traffic.
Practically, it may look like the connection or session has been dropped at that point. This can happen on existing packets in the original direction, but can also happen in the same way on return packets.
This is visible in the flow trace (as 'find a route') as the routing decision suddenly changes, but it is also visible on the session list (truncated output).
In this first snippet below, the original routing and its interfaces (43<>58) are shown. Note the serial code as well, as it is the session identifier in the session list and flow trace.
This session is not offloaded.
session info: proto=6 proto_state=01 duration=39 expire=3599 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=4
state=may_dirty synced
orgin->sink: org pre->post, reply pre->post dev=43->58/58->43 gwy=172.16.100.10/192.168.100.1
hook=pre dir=org act=noop 10.0.100.127:59843->172.16.100.10:443(0.0.0.0:0)
hook=post dir=reply act=noop 172.16.100.10:443->10.0.100.127:59843(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
dst_mac=00:09:0f:09:00:00
serial=00296968 tos=ff/ff app_list=0 app=0 url_cat=0
npu_state=0x040000
no_ofld_reason: non-npu-intf
The second session snipped shows the session got marked as dirty and that the interfaces (dev) have changed according to the new routing information.
Units changed from 43<>58 to 60<>0, but they are expected to stay the same.
session info: proto=6 proto_state=01 duration=54 expire=3599 timeout=3600 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=7
state=dirty may_dirty npu synced
orgin->sink: org pre->post, reply pre->post dev=60->0/0->60 gwy=0.0.0.0/192.168.254.124
hook=pre dir=org act=noop 10.0.100.127:59843->172.16.100.10:443(0.0.0.0:0)
hook=post dir=reply act=noop 172.16.100.10:443->10.0.100.127:59843(0.0.0.0:0)
pos/(before,after) 0/(0,0), 0/(0,0)
dst_mac=00:09:0f:09:00:00
serial=00296968 tos=ff/ff app_list=0 app=0 url_cat=0
npu_state=0x040000
npu info: flag=0x00/0x00, offload=0/0, ips_offload=0/0, epid=0/0, ipid=0/0, vlan=0x0000/0x0000
vlifid=0/0, vtag_in=0x0000/0x0000 in_npu=0/0, out_npu=0/0, fwd_en=0/0, qid=0/0
no_ofld_reason: dirty
If offloading is used on the interfaces, no such behavior is expected. In some cases, the capability of offloading by network design or hardware will not be granted. Work around this by eliminating the 'disturbing' routes, and redesign the setup to use offloading interfaces instead, or eventually use NAT between units/VDOMs to influence the routing decisions.
See the FortiGate hardware acceleration guide.
Related articles:
