Technical Tip: Understanding the net-device feature in FortiGate ADVPN implementation

Description

This article explains the ADVPN net-device feature and its beaviour.

Scope

FortiGate v7.0+

Solution

To configure ADVPN in v7.0+, the net-device feature should be disabled on the HUB FortiGate and enabled in all the spoke devices. 

This feature is configured under the IPsec VPN phase-1 configuration and helps in creating dynamic tunnel interfaces for the shortcut tunnels. Once a shortcut tunnel is created, a route is installed in the routing table and kernel routing table. This shortcut route is used for routing the traffic over the shortcut tunnel.  

config vpn ipsec phase1-interface 
    set net-device <enable/disabled>
end 

Example(ADVPN with BGP over loopback):

Spoke1 Public IP  =  10.9.10.109

Spoke 2 Public IP =  10.9.10.120

HUB Public IP      =  10.9.10.51

Src 172.17.1.1 --Spoke2--------------toHub1-Pri------------HUB----------------toHub1-Pri---------------Spoke1---------Dst 172.16.1.1

When the net-device is enabled on the Spoke2 firewall, the following routing information is observed: 

Initially, traffic is routed via the HUB FortiGate:

Spoke2 # get router info routing details 172.16.1.1

Routing table for VRF=0
Routing entry for 172.16.1.0/24
Known via "bgp", distance 200, metric 0, best
Last update 00:23:29 ago
* vrf 0 10.200.1.1 priority 1 

This is recursive via toHub1-Pri tunnel 10.9.10.51. 

A shortcut tunnel is created, and a dynamic shortcut route is installed in the routing table:

Spoke2 # exec ping 172.16.1.1
PING 172.16.1.1 (172.16.1.1): 56 data bytes
64 bytes from 172.16.1.1: icmp_seq=0 ttl=254 time=1.2 ms
64 bytes from 172.16.1.1: icmp_seq=1 ttl=254 time=0.6 ms
64 bytes from 172.16.1.1: icmp_seq=2 ttl=254 time=0.7 ms
64 bytes from 172.16.1.1: icmp_seq=3 ttl=254 time=0.7 ms
64 bytes from 172.16.1.1: icmp_seq=4 ttl=254 time=0.9 ms

 

Spoke2 # get router info routing details 172.16.1.1
Routing entry for 172.16.1.1/32
Known via "bgp", distance 200, metric 0, best
Last update 00:00:03 ago
* vrf 0 10.200.1.100 priority 1 

This is recursive via toHub1-Pri_0 tunnel 10.9.10.109.

Kernel route with the dynamic tunnel interface:

Spoke2 # get router info kernel | grep 172.16.1.1
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->172.16.1.1/32 pref=0.0.0.0 gwy=10.9.10.109 dev=26(toHub1-Pri_0)

Use the 'diagnose ip address list' command to provide additional information in cases where tunnel IPs are configured.

If the 'net-device' needs to be disabled on the spoke firewall, the traffic will still function normally.  

However, no shortcut route will be installed in the routing table. The kernel routing table will also not have the dynamic tunnel route.

Each dynamic tunnel is assigned a tunnel ID, represented as an IP address, which serves as the gateway in the route entry for that tunnel. This tunnel ID is used to identify the correct shortcut route. However, the IPsec tunnel monitor will still display the shortcut route, but it will not be installed in the routing table.

Note: 

SD-WAN ADVPN routing may not work correctly on the spoke where the net-device is disabled because routing information serves as one of the inputs for SD-WAN intelligence. Since dynamic shortcut interfaces are not added to the routing table, this can cause issues in selecting the outgoing interface for SD-WAN.

With 'net-device enable', each shortcut creates a dynamic interface named <phase-1 name>_<n>, and that whole string must stay within the 15-character interface-name limit.

A phase-1 name of 14–15 characters therefore prevents every shortcut from being created:

• 13 characters allows 16 shortcuts (the 17th fails).

• 12 characters allows 256 (the 257th fails).

Setting 'net-device disable' (FortiOS v6.2.2+) removes this by no longer counting the _<n> suffix toward the limit.

On ADVPN, the spoke (which creates shortcut interfaces and normally runs 'net-device enable') carries the constraint; the hub runs net-device disable and is not bound by it.

Symptom in the IKE debug:

could not create dialup name <name>_0, too long.

The tunnel list output after the shortcut is created when the 'net-device' is disabled:

Spoke2 # diagnose vpn tunnel list 
list all ipsec tunnel in vd 0
------------------------------------------------------
name=toHub1-Pri_0 ver=2 serial=6 10.9.10.120:0->10.9.10.109:0 tun_id=10.9.10.109 tun_id6=::10.9.10.109 status=up dst_mtu=1500 weight=1
bound_if=3 lgwy=static/1 tun=intf mode=dial_inst/3 encap=none/66216 options[102a8]=npu rgwy-chg frag-rfc run_state=0 role=primary accept_traffic=1 overlay_id=1

Routing table:

Spoke2 # get router info routing-table de 172.16.1.1

Routing table for VRF=0
Routing entry for 172.16.1.1/32
Known via "bgp", distance 200, metric 0, best
Last update 01:19:58 ago
* vrf 0 10.200.1.100 priority 1 (recursive via toHub1-Pri tunnel 10.9.10.109)

Kernel route after disabling the 'net-device' in the spoke firewall:

Spoke2 # get router info kernel | grep 172.16.1.1
tab=254 vf=0 scope=0 type=1 proto=11 prio=1 0.0.0.0/0.0.0.0/0->172.16.1.1/32 pref=0.0.0.0 gwy=10.9.10.109 dev=21(toHub1-Pri)

In the above route, although the outgoing interface to Hub1-Pri  belongs to the HUB firewall, the tunnel ID can identify the correct shortcut tunnel.

Related articles:

• Technical Tip: 'set net-device' new route-based IPsec logic

• Troubleshooting Tip: Error message 'unregister_netdevice: waiting for Dialup_VPN_Name to become free. Usage count = 1'

• Technical Tip: Unable to add IPsec Dial-up interface in an SD-WAN Zone

• Technical Tip: Reason behind why the net-device 'enable' is not supported on the HUB with SD-WAN