Fortigate - Traffic on backup WAN interfcae dropped

[jloureiro]

Hi all,

Scenario:

• Two site, each with a fortigate
• Site A and Site B connected via BGP
• Site B gets main internet access through a default route advertised by Site A
• Sibte B also as a Fortiextender (4G) for backup WAN with a Public IP (static route with higher Administrative Distance than BGP)
• Mail goal is remote host's to do VPN directly for Site B Fortigate, even when dafult route is being imported from the BGP
• Site B Fortigate 61F FortiOS 7.2.11

Setup is proven to be working as when I do a specific static route to my public IP I can reach site B Fortigate (ping and VPN).

I tried to acomplish this with Policy Based Route however traffic was being dropped by RPF, even with src-check disabled in the interface. For test purpose I did a PBR as less strict as possible.

config router policy
    edit 1
        set input-device "WAN-BACKUP-5G"
        set src "0.0.0.0/0.0.0.0"
        set dst "0.0.0.0/0.0.0.0"
        set gateway <Fortiextender next-hop>
        set output-device "WAN-BACKUP-5G"
    next
end

After some troubleshoot I can see session is being created for inboud ping, but no reply packets counted:

session info: proto=1 proto_state=00 duration=2 expire=57 timeout=0 flags=00000000 socktype=0 sockport=0 av_idx=0 use=3 origin-shaper= reply-shaper= per_ip_shaper= class_id=0 ha_id=0 policy_dir=0 tunnel=/ vlan_cos=0/0 state=log local may_dirty statistic(bytes/packets/allow_err): org=60/1/1 reply=0/0/0 tuples=2 tx speed(Bps/kbps): 29/0 rx speed(Bps/kbps): 0/0 orgin->sink: org pre->in, reply out->post dev=26->18/18->26 gwy=193.126.22.140/0.0.0.0 hook=pre dir=org act=noop "my_public_IP":1->"FEXT_public_IP":8(0.0.0.0:0) hook=post dir=reply act=noop "FEXT_public_IP":1->"my_public_IP":0(0.0.0.0:0) src_mac="MAC" misc=0 policy_id=1 pol_uuid_idx=722 auth_info=0 chk_client_info=0 vd=0 serial=05172b18 tos=ff/ff app_list=0 app=0 url_cat=0 rpdb_link_id=00000000 ngfwid=n/a npu_state=00000000 no_ofld_reason: local

Thansk!

[AEK]

Hi Joao

No need for policy route. Just add a route like this:

• Destination: Public IP of the remote VPN server
• Device: WAN-BKP-5G
• Gateway: x.x.x.x

Then the VPN connection will automatically go through the specified gateway.

[jloureiro]

Hi AEK, thanks for your reply.

I tried that at the beginning, but the connection doesn’t go through. In the logs, I can see the reserve path check failing — I assume it’s because it tries to go “out” through the main WAN, which has its route installed in the routing table.

09:47:06 75 vd-root:0 received a packet(proto=6, "my_public_ip"->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 4008080042, ack 0, win 65535
09:47:06 75 allocate a new session-05a833f3
09:47:06 75 in-[WAN-BACKUP-5G], out-[]
09:47:06 75 len=0
09:47:06 75 result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000
09:47:06 75 reverse path check fail, drop
09:47:06 75 trace
09:47:07 76 vd-root:0 received a packet(proto=6, "my_public_ip":52569->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 4008080042, ack 0, win 65535
09:47:07 76 allocate a new session-05a834a1
09:47:07 76 in-[WAN-BACKUP-5G], out-[]
09:47:07 76 len=0
09:47:07 76 result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000
09:47:07 76 reverse path check fail, drop
09:47:07 76 trace
09:47:08 77 vd-root:0 received a packet(proto=6, "my_public_ip":52569->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 4008080042, ack 0, win 65535
09:47:08 77 allocate a new session-05a8350d
09:47:08 77 in-[WAN-BACKUP-5G], out-[]
09:47:08 77 len=0
09:47:08 77 result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000
09:47:08 77 reverse path check fail, drop
09:47:08 77 trace

When I disable src-check in the interface:

09:55:01 87 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:01 87 allocate a new session-05a8ae64
09:55:01 87 in-[WAN-BACKUP-5G], out-[]
09:55:01 87 len=0
09:55:01 87 result: skb_flags-02000000, vid-0, ret-no-match, act-accept, flag-00000000
09:55:01 87 find a route: flag=80000000 gw-"vpn_gateway" via root
09:55:01 87 in-[WAN-BACKUP-5G], out-[], skb_flags-02000000, vid-0
09:55:01 87 gnum-100017, check-ffffffbffc02bd34
09:55:01 87 after check: ret-no-match, act-accept, flag-00000000, flag2-00000000
09:55:01 87 in-[WAN-BACKUP-5G], out-[], skb_flags-02000000, vid-0
09:55:01 87 gnum-100011, check-ffffffbffc02cd00
09:55:01 87 after check: ret-no-match, act-drop, flag-00000000, flag2-00000000
09:55:01 87 gnum-100001, check-ffffffbffc02bd34
09:55:01 87 after check: ret-no-match, act-accept, flag-00000000, flag2-00000000
09:55:01 87 gnum-10000e, check-ffffffbffc02bd34
09:55:01 87 checked gnum-10000e policy-4294967295, ret-no-match, act-accept
09:55:01 87 checked gnum-10000e policy-4294967295, ret-no-match, act-accept
09:55:01 87 checked gnum-10000e policy-4294967295, ret-no-match, act-accept
09:55:01 87 checked gnum-10000e policy-4294967295, ret-no-match, act-accept
09:55:01 87 checked gnum-10000e policy-4294967295, ret-matched, act-accept
09:55:01 87 policy-4294967295 is matched, act-accept
09:55:01 87 gnum-10000e check result: ret-matched, act-accept, flag-00000001, flag2-00000000
09:55:01 87 after check: ret-matched, act-accept, flag-00000001, flag2-00000000
09:55:01 88 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:01 88 Find an existing session, id-05a8ae64, original direction
09:55:02 89 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:02 89 Find an existing session, id-05a8ae64, original direction
09:55:03 90 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:03 90 Find an existing session, id-05a8ae64, original direction
09:55:04 91 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:04 91 Find an existing session, id-05a8ae64, original direction
09:55:05 92 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:05 92 Find an existing session, id-05a8ae64, original direction
09:55:08 93 vd-root:0 received a packet(proto=6, "my_public_ip":52571->"vpn_gateway":10069) tun_id=0.0.0.0 from WAN-BACKUP-5G. flag [S], seq 13269040, ack 0, win 65535
09:55:08 93 Find an existing session, id-05a8ae64, original direction

Thanks

[sjoshi]

your issue is because of RPF failure

can you share your routing table:

get router info routing-table all

[jloureiro]

Yes I think that is the issue, that's why I tried it with the PBR but for some reason the traffic seems not to match the policy.

Routing table:

SBG-FW-LB-MGMT-02 $ get router info routing-table details "my_public_ip"

Routing table for VRF=0
Routing entry for 0.0.0.0/0
  Known via "bgp", distance 20, metric 0, best
  Last update 5d23h51m ago
  * vrf 0 A.B.C.D priority 1 (recursive is directly connected, INTERLIG-ALTICE)

Routing entry for 0.0.0.0/0
  Known via "static", distance 240, metric 0
    vrf 0 "interface_next_hop", via WAN-BACKUP-5G

[syordanov]

Hi João,

Based on the output from the routing table above, you have two default routes , one BGP with AD of 20 and one static with AD of 240. When you have 2 route for the same destination with different AD, FortiOS will install the one with lowest AD , so the the static one is inactive.
You can verify this using the commands bellow :

#get router info routing-table all

#get router info routing-table database

BR,

Fortinet

[jloureiro]

Hi syordanov,

Thanks for your reply. I am aware of that, however I was trying to find a way for me to connect to the VPN which is listening in the backup wan interface.

[AEK]

@syordanov is right, you should set the distance of the static route to something below 20, so the static route will be selected when you try connect to the remote VPN server.

Once done it should work as you are expecting.

[jloureiro]

It will connect, but it will be de default route for everyone else, while the goal is:
- Default must be imported by BGP from site A

- VPN should still be able on the backup interface on site B

[AEK]

You are right, I didn't notice that in the output. What I mean is that you need a route poining on a single IP as described in my first post and make sure its AD is lower than any existing similar route. Also delete the policy route that you shown in your first post.