Technical Tip: NPU offloaded session not failing over according to SD-WAN rule SLA

[sagha]

Description
This article explains the scenario where if session established on FortiGate and offloaded to NPU, if SLA of an SD-WAN rule changes, the session sticks to the same interface.

Explanation.

Session is offloaded to NPU.

FGT # diag sys session list
session info: proto=1 proto_state=00 duration=117 expire=23 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=4
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=Vpn_2/ vlan_cos=0/255
state=may_dirty npu
statistic(bytes/packets/allow_err): org=120/2/1 reply=120/2/1 tuples=2
tx speed(Bps/kbps): 1/0 rx speed(Bps/kbps): 1/0
orgin->sink: org pre->post, reply pre->post dev=35->40/40->35 gwy=2.2.2.2/192.168.100.110
hook=pre dir=org act=noop 192.168.100.110:1->192.168.1.111:8(0.0.0.0:0)
hook=post dir=reply act=noop 192.168.1.111:1->192.168.100.110:0(0.0.0.0:0)
misc=0 policy_id=6 auth_info=0 chk_client_info=0 vd=0
serial=01702715 tos=ff/ff app_list=0 app=0 url_cat=0
vwl_mbr_seq=4 vwl_service_id=1
rpdb_link_id=ff000001 ngfwid=n/a
dd_type=0 dd_mode=0
npu_state=0x3000c00
npu info: flag=0x82/0x81, offload=8/8, ips_offload=0/0, epid=67/88, ipid=88/67, vlan=0x0000/0x0000
vlifid=88/249, vtag_in=0x0000/0x0000 in_npu=1/1, out_npu=1/1, fwd_en=0/0, qid=0/1
total session 1

### Introduced latency ###

FGT # diag sys virtual-wan-link health-check test_sla
Health Check(test_sla):
Seq(3): state(alive), packet-loss(0.000%) latency(0.773), jitter(0.120) sla_map=0x1
Seq(4): state(alive), packet-loss(0.000%) latency(97.195), jitter(26.800) sla_map=0x0        <----- Latency added.

FGT # diag sys virtual-wan-link service 1
Service(1): Address Mode(IPV4) flags=0x0
TOS(0x0/0x0), Protocol(0: 1->65535), Mode(priority), link-cost-factor(latency), link-cost-threshold(10), health-check(test_sla)
Service role: standalone
Member sub interface(2):
Members(2):
1: Seq_num(3), alive, latency: 0.767, selected                                                      <----- Order changes, member 3 is first one which means traffic should switch over.
2: Seq_num(4), alive, latency: 90.516, selected
Src address(1):
0.0.0.0-255.255.255.255

Despite the change in order of the interface in SD-WAN rule as per SLA, the traffic sticks to the same interface with session offloaded.

FGT # diag sys session list
session info: proto=1 proto_state=00 duration=134 expire=47 timeout=0 flags=00000000 sockflag=00000000 sockport=0 av_idx=0 use=4
origin-shaper=
reply-shaper=
per_ip_shaper=
class_id=0 ha_id=0 policy_dir=0 tunnel=Vpn_2/ vlan_cos=0/255
state=may_dirty npu
statistic(bytes/packets/allow_err): org=120/2/1 reply=120/2/1 tuples=2
tx speed(Bps/kbps): 1/0 rx speed(Bps/kbps): 1/0
orgin->sink: org pre->post, reply pre->post dev=35->40/40->35 gwy=2.2.2.2/192.168.100.110
hook=pre dir=org act=noop 192.168.100.110:1->192.168.1.111:8(0.0.0.0:0)
hook=post dir=reply act=noop 192.168.1.111:1->192.168.100.110:0(0.0.0.0:0)
misc=0 policy_id=6 auth_info=0 chk_client_info=0 vd=0
serial=01702715 tos=ff/ff app_list=0 app=0 url_cat=0
vwl_mbr_seq=4 vwl_service_id=1                                                                      <----- Not changing to interface.
rpdb_link_id=ff000001 ngfwid=n/a
dd_type=0 dd_mode=0
npu_state=0x3000c00
npu info: flag=0x82/0x81, offload=8/8, ips_offload=0/0, epid=67/88, ipid=88/67, vlan=0x0000/0x0000
vlifid=88/249, vtag_in=0x0000/0x0000 in_npu=1/1, out_npu=1/1, fwd_en=0/0, qid=0/1
total session 1

Solution
In order change this behavior; it is required to enable the following so the best route is selected once SD-WAN rule SLA has changed:

# config system settings
    set auxiliary-session enable
end

When the return traffic reaches the same port as it left, the following applies.

1) Auxiliary sessions disabled.

The reply to the client egresses on the original incoming interface.

2) Auxiliary sessions enabled.

The reply to the client egresses on the best route in the routing table:
- If the best route is WAN1, then it will egress on WAN1.
- If the best route is WAN2, then it will egress on WAN2.

Related document.
https://docs.fortinet.com/document/fortigate/6.2.0/new-features/14295/controlling-return-path-with-a...