Help
Support Forum
The Forums are a place to find answers on a range of Fortinet products from peers and product experts.
POMA
New Contributor

Created on ‎11-22-2023 07:36 AM

strange routing behaviour (sslvpn > site HQ > ipsec site Y)

Hi, having a strange issue on a basic setup...  

 

Basic setup

site HQ: Fortigate 100F R7.4.1.  Internal IP 192.168.17.252/24

site X: Fortigate 60F R7.4.1.  Internal IP 10.254.28.1/24

site Y,Z, ...: Fortigate 60F R7.4.1.  Other subnets (so, no overlaps)

 

site HQ

  • IPSEC tnl1_siteX exist (HQ WAN1 -> site X WAN1)
  • IPSEC tnl2_SiteX exist (HQ WAN2 -> site X WAN1)
  • same towards site Y, Z, ... --> tnl1_siteY + tnl2_siteY, tnl2_SiteZ, ...
  • static route pointing to 10.254.28.0 (site X) via tnl1 + tnl2 (same distance/priority).  Als tried with the route pointing to the SDWAN zone "RemoteSites" (which has all the tunnels towards other sites in it)
  • policy present from SSLVPN towards "RemoteSites" (= the SDWAN zone with the tunnels)
  • SDWAN rules present: from "all", towards 10.254.28.0/24 via tnl1_SiteX + tnl2_SiteY

site X

  • IPSEC tnl1_HQ exist 
  • IPSEC tnl2_HQ exist
  • static route towards 10.212.130.0/24 (= ssl of HQ) points to RemoteSites (= SDWAN zone with all the tunnels in it)
  • SDWAN rules exist towards HQ (from all, destination LAN of HQ + 10.212.130.0/24), via tnl1_HQ + tnl2_HQ (tnl1 = preferred)
  • policies exist (from RemoteSites towards LAN)

 

Behavior I see

  • VPN connection towards HQ.  Gives me IP 10.212.130.1 (= OK).  Route print of my laptop shows 10.254.28.0/24 exists (via VPN)
  • Ping from laptop towards 10.254.28.1 = ok (reply).  
  • SSH, HTTPS, ... towards 10.254.28.1 does not work.  
    • doing a diag debug on HQ -> shows the packet from 10.212.130.1 (vpn address) towards 10.254.28.1 is sent over the correct IPSEC tunnel
    • doing a diag debug on siteX -> shows the packet is received, but the return traffic is going to a wrong tunnel (tnl1_SiteY instead of tnl1_HQ)

Strange thing, I had this issue with 3 of 4 remote sites.  

I tried giving a different IP (or adding a secondary IP) to SiteY and for some IP's the problem is resolved (so https, ssh, ... work).  When reverting back to the original situation, so SiteY has only 1 IP (the original one), it kept working.  

 

So, in short...  Ping works.  SSH, https, ... do not. I see the packet is received at the correct site, but the return traffic is using the wrong tunnel.

 

Workaroud = the static route on SiteX, instead of pointing to the SDWAN zone I have to point to tnl1_HQ + tnl2_HQ 

 

I'm not really sur if this i a know issue in R7.4.1 (there are similar issues in the doc, ex bug id 961796 - they are related to an incorrect routing of the response packet).

 

 

Some relevant code & diag debug output.

ps. for this post, I changed the names to tnl1_HQ, tnl1_siteX, ... (in real life it's using a different name - I tried to replace the info in the code below) - if it's too confusing I'll change my post.

 

I checked the policies, routes, ... via GUI (visual + using the policy+route lookup buttons) + CLI.  They seem fine (this is a setup I'm using for multiple clients, so I know what has to be configured).

 

HQ

 

 

 

 

config router static
edit 4
        set dst 10.254.28.0 255.255.255.0
        set distance 1
        set device "tnl1_SiteX"
        set comment "SiteX: LAN"
    next
end

config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "Internet"
        next
        edit "RemoteSites"
        next
    end
    config members
        edit 1
            set interface "wan1"
            set zone "Internet"
        next
        edit 2
            set interface "tnl1_Genk"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 3
            set interface "tnl2_Genk"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 4
            set interface "tnl1_Korten"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 7
            set interface "tnl2_Korten"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 6
            set interface "tnl1_Neeroet"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 8
            set interface "tnl2_Neeroet"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 9
            set interface "tnl1_siteX"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 10
            set interface "tnl2_siteX"
            set zone "RemoteSites"
            set source 192.168.17.252
        next
        edit 11
            set interface "wan2"
            set zone "Internet"
            set gateway 89.41.54.158
        next
        edit 5
            set source 192.168.17.252
        next
    end
    config health-check
        edit "SLA_Genk"
            set server "10.254.30.1"
            set update-static-route disable
            set members 2 3
        next
        edit "SLA_Ravels"
            set server "10.254.28.1"
            set update-static-route disable
            set members 9 10
        next
        edit "SLA_Kortenaken"
            set server "192.168.16.252"
            set update-static-route disable
            set members 4 7
        next
        edit "SLA_Internet"
            set server "8.8.8.8" "1.1.1.1"
            set members 1 11
        next
        edit "SLA_Neeroeteren"
            set server "10.254.25.1"
            set update-static-route disable
            set members 6 8
        next
    end
    config service
        edit 1
            set name "Genk"
            set dst "LAN_Genk"
            set src "LAN_Tessenderlo" "SSLVPN_Tessenderlo"
            set priority-members 2 3
        next
        edit 2
            set name "Neeroeteren"
            set dst "LAN_Neeroeteren"
            set src "LAN_Tessenderlo" "SSLVPN_Tessenderlo"
            set priority-members 6 8
        next
        edit 3
            set name "siteX"
            set dst "LANsiteX" "test"
            set src "HQ" "SSLVPN_HQ-VPN"
            set priority-members 9 10
        next
        edit 4
            set name "Kortenaken"
            set dst "LAN_Kortaken"
            set src "LAN_Tessenderlo" "SSLVPN_Tessenderlo"
            set priority-members 4 7
        next
        edit 5
            set name "Guest_2_Internet"
            set dst "all"
            set src "Guest_Tessenerlo"
            set priority-members 11 1
        next
        edit 6
            set name "Internet"
            set dst "all"
            set src "all"
            set priority-members 1 11
        next
    end
end

 

 

 

 

 

site X

 

 

 

 

config router static
    edit 8
        set dst 10.212.130.0 255.255.255.0
        set distance 1
        set comment "HQ: SSLVPN BKM"
        set sdwan-zone "RemoteSites"
    next
end

config system sdwan
    set status enable
    config zone
        edit "virtual-wan-link"
        next
        edit "Internet"
        next
        edit "RemoteSites"
        next
    end
    config members
        edit 1
            set interface "wan1"
            set zone "Internet"
            set gateway 89.41.54.166
        next
        edit 2
            set interface "tnl1_HQ"
            set zone "RemoteSites"
            set source 10.254.28.1
        next
        edit 3
            set interface "tnl2_HQ"
            set zone "RemoteSites"
            set source 10.254.28.1
        next
        edit 4
            set interface "tnl1_Korten"
            set zone "RemoteSites"
            set source 10.254.28.1
        next
        edit 5
            set interface "tnl1_Neeroet"
            set zone "RemoteSites"
            set source 10.254.28.1
        next
        edit 6
            set interface "tnl1_Genk"
            set zone "RemoteSites"
            set source 10.254.28.1
        next
    end
    config health-check
        edit "SLA_Tessenderlo"
            set server "192.168.17.252"
            set update-static-route disable
            set members 2 3
        next
        edit "SLA_Genk"
            set server "10.254.30.250" "10.254.30.1"
            set update-static-route disable
            set members 6
        next
        edit "SLA_Neeroeteren"
            set server "10.254.25.250" "10.254.25.1"
            set update-static-route disable
            set members 5
        next
        edit "SLA_Kortenaken"
            set server "192.168.16.1" "192.168.16.252"
            set update-static-route disable
            set members 4
        next
        edit "SLA_Internet"
            set server "8.8.8.8" "1.1.1.1"
            set update-static-route disable
            set members 1
        next
    end
    config service
        edit 1
            set name "Genk"
            set dst "LAN_Genk"
            set src "LAN_Ravels"
            set priority-members 6
        next
        edit 2
            set name "Neeroeteren"
            set dst "LAN_Neeroeteren"
            set src "LAN_Ravels"
            set priority-members 5
        next
        edit 3
            set name "Kortenaken"
            set dst "LAN_Kortenaken"
            set src "LAN_Ravels"
            set priority-members 4
        next
        edit 4
            set name "HQ"
            set dst "LAN_HQ" "Servers_Tessenderlo" "SSLVPN_HQ"
            set src "LAN_siteX"
            set priority-members 2 3
        next
        edit 5
            set name "Internet"
            set dst "all"
            set src "all"
            set priority-members 1
        next
    end
end

TUBRAVFW01 (sdwan) #

 

 

 

 

 

 

Diag debug on siteX of ssh that's going wrong

id 221 = the packet arriving (from tnl1_Tessend => I'm using tnl1_HQ in this explanation)

id 222 = the returning packet (going to 10.212.130.1, but using tnl1_Neeroet... = the wrong tunnel)

 

 

 

 

TUBRAVFW01 # diag debug flow filter addr 10.212.130.1

TUBRAVFW01 # diagnose debug flow trace start 100

TUBRAVFW01 # di deb en

TUBRAVFW01 # id=65308 trace_id=221 func=print_pkt_detail line=5832 msg="vd-root:0 received a packet(proto=6, 10.212.130.1:59596->10.254.28.1:22) tun_id=81.241.240.192 from tnl1_Tessend. flag [S], seq 2972104827, ack 0, win 64896"
id=65308 trace_id=221 func=init_ip_session_common line=6017 msg="allocate a new session-000080ff, tun_id=81.241.240.192"
id=65308 trace_id=221 func=vf_ip_route_input_common line=2611 msg="find a route: flag=80000000 gw-10.254.28.1 via root"
id=65308 trace_id=221 func=__iprope_tree_check line=539 msg="gnum-100004, use addr/intf hash, len=3"
id=65308 trace_id=222 func=print_pkt_detail line=5832 msg="vd-root:0 received a packet(proto=6, 10.254.28.1:22->10.212.130.1:59596) tun_id=0.0.0.0 from local. flag [S.], seq 1642619526, ack 2972104828, win 65535"
id=65308 trace_id=222 func=resolve_ip_tuple_fast line=5920 msg="Find an existing session, id-000080ff, reply direction"
id=65308 trace_id=222 func=ip_session_core_in line=6549 msg="dir-1, tun_id=81.241.240.192"
id=65308 trace_id=222 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface tnl1_Neeroet, tun_id=81.241.240.192"
id=65308 trace_id=222 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel tnl1_Neeroet, tun_id=89.41.54.169, vrf 0"
id=65308 trace_id=222 func=esp_output4 line=896 msg="IPsec encrypt/auth"
id=65308 trace_id=222 func=ipsec_output_finish line=629 msg="send to 89.41.54.166 via intf-wan1"

 

 

 

 

 Diag debug on SiteX of ping (which works fine)

id 230 = ping arriving to siteX (from tnl1_Tessend => I'm using tnl1_HQ in this explanation)

id 231 = reply going to HQ (sent back via tnl1_Tessend = tnl1_HQ)

 

 

 

 

TUBRAVFW01 # diag debug flow filter addr 10.212.130.1

TUBRAVFW01 # diagnose debug flow trace start 100

TUBRAVFW01 # di deb en

TUBRAVFW01 # id=65308 trace_id=230 func=print_pkt_detail line=5832 msg="vd-root:0 received a packet(proto=1, 10.212.130.1:1->10.254.28.1:2048) tun_id=81.241.240.192 from tnl1_Tessend. type=8, code=0, id=1, seq=86."
id=65308 trace_id=230 func=init_ip_session_common line=6017 msg="allocate a new session-0000850f, tun_id=81.241.240.192"
id=65308 trace_id=230 func=vf_ip_route_input_common line=2611 msg="find a route: flag=80000000 gw-10.254.28.1 via root"
id=65308 trace_id=230 func=__iprope_tree_check line=539 msg="gnum-100004, use addr/intf hash, len=3"
id=65308 trace_id=231 func=print_pkt_detail line=5832 msg="vd-root:0 received a packet(proto=1, 10.254.28.1:1->10.212.130.1:0) tun_id=0.0.0.0 from local. type=0, code=0, id=1, seq=86."
id=65308 trace_id=231 func=resolve_ip_tuple_fast line=5920 msg="Find an existing session, id-0000850f, reply direction"
id=65308 trace_id=231 func=ip_session_core_in line=6549 msg="dir-1, tun_id=81.241.240.192"
id=65308 trace_id=231 func=ipsecdev_hard_start_xmit line=669 msg="enter IPSec interface tnl1_Tessend, tun_id=81.241.240.192"
id=65308 trace_id=231 func=_do_ipsecdev_hard_start_xmit line=229 msg="output to IPSec tunnel tnl1_Tessend, tun_id=81.241.240.192, vrf 0"
id=65308 trace_id=231 func=esp_output4 line=896 msg="IPsec encrypt/auth"
id=65308 trace_id=231 func=ipsec_output_finish line=629 msg="send to 89.41.54.166 via intf-wan1"

 

 

 

 

 

routes on HQ

10.254.28.0 shows all the tunnels (tnl1_Ravels + tnl2_Ravels are the tnl1_X  I refer too).   There is no more specific route defined for this subnet

 

 

 

 

TUBTESFW01-NEW # get router info routing-table all 
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       V - BGP VPNv4
       * - candidate default

Routing table for VRF=0
S*      0.0.0.0/0 [1/0] via 89.41.54.158, wan2, [1/0]
                  [1/0] via 192.168.128.1, wan1, [1/0]
S       3.124.165.251/32 [10/0] via 192.168.17.1, lan, [1/0]
S       10.254.25.0/24 [1/0] via tnl1_Neeroet tunnel 89.41.54.169, [1/0]
S       10.254.28.0/24 [1/0] via tnl2_Ravels tunnel 10.0.0.4, [1/0]
                       [1/0] via tnl1_Ravels tunnel 89.41.54.165, [1/0]
                       [1/0] via 192.168.17.1, lan, [1/0]

 

 

 

 

 routes on site X

10.212.130.0/24 via all the tunnels (tnl1_Tessend + tnl2_Tessend are the tnl1_HQ + tnl2_HQ I refer to in this post)

 

 

 

 

Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       V - BGP VPNv4
       * - candidate default

Routing table for VRF=0
S*      0.0.0.0/0 [1/0] via 89.41.54.166, wan1, [1/0]
S       3.124.165.251/32 [10/0] via 10.254.28.250, internal, [1/0]
S       10.212.130.0/24 [1/0] via tnl1_Tessend tunnel 81.241.240.192, [1/0]
                        [1/0] via tnl1_Genk tunnel 81.241.240.193, [1/0]
                        [1/0] via tnl2_Tessend tunnel 89.41.54.157, [1/0]
                        [1/0] via tnl1_Neeroet tunnel 89.41.54.169, [1/0]
                        [1/0] via tnl1_Korten tunnel 89.41.54.173, [1/0]

 

 

 

 

 

Labels:
1078
0 Kudos
3 REPLIES 3
AEK
SuperUser
SuperUser

Created on ‎11-23-2023 07:14 AM

Hello

Is there any related policy route on siteX?

Also please share the following commands:

  • get sys settings | grep asymroute
  • get sys settings | grep auxiliary
AEK
AEK
1047
POMA
New Contributor

Created on ‎11-24-2023 12:25 AM Edited on ‎11-24-2023 12:28 AM

For SiteX, the only policy routes are the SDWAN rules.  the output is below.

The relevant line is id=2131361796, with 10.212.130.0 - .255 as destination

Auxiliary-session is disabled (but tried with enabling it before)

 

 

TUBRAVFW01 # get firewall proute 
list route policy info(vf=root):

id=2131361793(0x7f0a0001) vwl_service=1(Genk) vwl_mbr_seq=6 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0(any) 
path(1): oif=26(tnl1_Genk)
source(1): 10.254.28.0-10.254.28.255 
destination(1): 10.254.30.0-10.254.30.255 
hit_count=68649 last_used=2023-11-24 09:20:07

id=2131361794(0x7f0a0002) vwl_service=2(Neeroeteren) vwl_mbr_seq=5 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0(any) 
path(1): oif=27(tnl1_Neeroet)
source(1): 10.254.28.0-10.254.28.255 
destination(1): 10.254.25.0-10.254.25.255 
hit_count=67911 last_used=2023-11-24 09:20:07

id=2131361795(0x7f0a0003) vwl_service=3(Kortenaken) vwl_mbr_seq=4 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0(any) 
path(1): oif=28(tnl1_Korten)
source(1): 10.254.28.0-10.254.28.255 
destination(1): 192.168.16.0-192.168.16.255 
hit_count=68631 last_used=2023-11-24 09:20:07

id=2131361796(0x7f0a0004) vwl_service=4(Tessenderlo) vwl_mbr_seq=2 3 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0(any) 
path(2): oif=24(tnl1_Tessend), oif=25(tnl2_Tessend)
source(1): 10.254.28.0-10.254.28.255 
destination(4): 
        192.168.21.0-192.168.21.255 
        10.212.130.0-10.212.130.255 
        192.168.50.0-192.168.50.255 
        192.168.17.0-192.168.17.255 

hit_count=6000 last_used=2023-11-24 09:20:07

id=2131361797(0x7f0a0005) vwl_service=5(Internet) vwl_mbr_seq=1 dscp_tag=0xfc 0xfc flags=0x0 tos=0x00 tos_mask=0x00 protocol=0 port=src(0->0):dst(0->0) iif=0(any) 
path(1): oif=5(wan1)
source(1): 0.0.0.0-255.255.255.255 
destination(1): 0.0.0.0-255.255.255.255 
hit_count=87566 last_used=2023-11-24 09:20:10



TUBRAVFW01 # get sys settings | grep asymroute
asymroute           : disable 
asymroute-icmp      : disable 
asymroute6          : disable 
asymroute6-icmp     : disable 

TUBRAVFW01 # get sys settings | grep auxiliary
auxiliary-session   : disable 

TUBRAVFW01 #

 

 

ps. I realize all the outputs are maybe confusing due to other IPSEC tunnels (different naming, ...).  I'm going to make a similar setup in our lab environment so I can minimize the code...

 

1013
0 Kudos
ezhupa
Staff
Staff

Created on ‎12-01-2023 07:37 AM

Hello POMA,

 

>I'm not really sur if this i a know issue in R7.4.1 (there are similar issues in the doc, ex bug id 961796 - they are related to an incorrect routing of the response packet).

Most likely you are indeed matching the internal ticket you mentioned. The setup is very similar to other setups linked to that internal ticket and basically, the issue is related to the incorrect routing of response packet when activating HTTPS access on a SDWAN member interface.

The internal ticket is still under investigation and we can not share more information regarding it since it is internal information. 
Feel free to open a ticket in case you have a valid support contract for your devices. 

Thank you for sharing your issue and the workarounds you have found. 

896
0 Kudos
Labels
Top Kudoed Authors
View all
Broad. Integrated. Automated.

The Fortinet Security Fabric brings together the concepts of convergence and consolidation to provide comprehensive cybersecurity protection for all users, devices, and applications and across all network edges.

Social Media
Security Research
Company
News & Articles
Contact Us

Copyright 2024 Fortinet, Inc. All Rights Reserved.