Description
This article describes the expected behavior for FortiOS v6.2.3 when traffic egresses on one interface and asymmetrically returns on a different interface, and it also describes how to enable auxiliary sessions to address this scenario. This is a common case when ECMP (equal-cost multipath) routing or SD-WAN is utilized,
Note: This behavior is specific to FortiOS 6.2.3 only. FortiOS 6.0 and earlier, v6.2.0 to v6.2.2, and v6.2.4 onward all handle asymmetric traffic differently (see Scenario 2 here: Controlling return path with auxiliary session).
Scope
FortiOS v6.2.3 only.
Solution
FortiGates configured for ECMP and/or SD-WAN can have multiple paths to reach a given remote network destination, and when the FortiGate receives traffic from a client, it will perform a route lookup and select one of the ECMP routes as the outgoing path. However, in some cases, reply traffic can arrive back on the FortiGate using a different ECMP interface than was used in the originating direction. This is referred to as asymmetric traffic flow.
In FortiOS v6.2.3 specifically, the FortiGate kernel handled this flow differently from other versions before or since. Consider the following debug flow output for a TCP session between a client and a server, which is based on the topology diagram shown above:
- Client 10.1.0.3 sends TCP SYN to Server 10.2.0.4. Packet arrives on FortiGate port2 and is routed out via port3:
id=20085 trace_id=13 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=6, 10.1.0.3:51156->10.2.0.4:5000) from port2. flag [S], seq 1490277004, ack 0, win 29200"
id=20085 trace_id=13 func=init_ip_session_common line=5666 msg="allocate a new session-0000002b"
id=20085 trace_id=13 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-198.51.100.14 via port3"
id=20085 trace_id=13 func=fw_forward_handler line=771 msg="Allowed by Policy-1:"
- Server 10.2.0.4 receives the packet and sends a SYN/ACK in response. The packet is routed back towards the FortiGate's port4 interface, rather than port3. The FortiGate matches the packet to the existing session, then updates the session to allow reply traffic from port4 towards port2:
id=20085 trace_id=14 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=6, 10.2.0.4:5000->10.1.0.3:51156) from port4. flag [S.], seq 3050259070, ack 1490277005, win 28960"
id=20085 trace_id=14 func=resolve_ip_tuple_fast line=5581 msg="Find an existing session, id-0000002b, reply direction"
id=20085 trace_id=14 func=vf_ip_route_input_common line=2596 msg="find a route: flag=00000000 gw-10.1.0.3 via port2"
id=20085 trace_id=14 func=npu_handle_session44 line=1139 msg="Trying to offloading session from port4 to port2, skb.npu_flag=00000400 ses.state=00000280 ses.npu_state=0x00040000"
id=20085 trace_id=14 func=fw_forward_dirty_handler line=449 msg="state=00000280, state2=00000000, npu_state=00040000"
- Client 10.1.0.3 sends a TCP ACK to complete the TCP three-way handshake. However, the FortiGate drops the packet because the previous SYN/ACK caused the outgoing interface for the session to be changed/relearned. When the client attempts to retransmit the ACK, the packet is dropped since the session entry no longer exists:
id=20085 trace_id=15 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=6, 10.1.0.3:51156->10.2.0.4:5000) from port2. flag [.], seq 1490277005, ack 3050259071, win 229"
id=20085 trace_id=15 func=resolve_ip_tuple_fast line=5581 msg="Find an existing session, id-0000002b, original direction"
id=20085 trace_id=15 func=ip_session_core_in line=6275 msg="outgoing dev changed:6->5 dir=original, drop"
id=20085 trace_id=15 func=print_pkt_detail line=5501 msg="vd-root:0 received a packet(proto=6, 10.1.0.3:51156->10.2.0.4:5000) from port2. flag [.], seq 1490277005, ack 3050259071, win 229"
id=20085 trace_id=15 func=vf_ip_route_input_common line=2596 msg="find a route: flag=04000000 gw-198.51.100.14 via port3"
id=20085 trace_id=15 func=fw_forward_dirty_handler line=385 msg="no session matched"
In FortiOS v6.2.3, this issue can be addressed by enabling the auxiliary-session feature (configured on a per-VDOM basis). This allows the FortiGate to create secondary sessions (aka auxiliary or reflect sessions) that can allow this asymmetric traffic to flow successfully:
config system settings
set auxiliary-session [ enable | disable ]
end
Note that in FortiOS v6.2.4 and later, this behavior has changed such that traffic will flow successfully even when auxiliary-session is disabled. Instead of dropping traffic, the FortiGate will instead continuously dirty the session whenever the outgoing interface changes. This allows traffic to flow, but it has the downside of preventing hardware-offloading from being activated for that particular session. Enabling auxiliary-session will correct this and allow the FortiGate to offload this asymmetric traffic flow successfully.
Related documents:
Controlling return path with auxiliary session
Changes in CLI Defaults
