VRF Leaking 6.4.5 with more the 2 VRFs

[virg9454]

FortiGate 60F FortiOS 6.4.5

I am trying to figure out how to build the npu0_vlink to be able to create VRF-Leaking with up to 5 VRFs.   I followed the instructions on https://docs.fortinet.com/document/fortigate/6.4.0/new-features/834664/route-leaking-between-vrfs, but to keeps things simple I am using static routes to add the routes, which works fine.  With 2 VRF.  But now I need to allow VRF 30 to also access VRF 10, and VRF 3 to access VRF 5.   I create new npu0_vlink2 and so one, but they don't seem to work if I  do a diag sys session list, it looks like it has a problem getting a mac for the remote side of the npu link.   

This works!

edit "npu0_vlink0"
        set vdom "root"
        set vrf 10
        set ip 172.16.201.1 255.255.255.0
        set allowaccess ping https ssh snmp http
    next
    edit "npu0_vlink1"
        set vdom "root"
        set vrf 20
        set ip 172.16.201.2 255.255.255.0
        set allowaccess ping https ssh snmp http telnet
    next
end

The below does not work!

edit "npu0_vlink3" set vdom "root" set vrf 3 set ip 172.16.201.3 255.255.255.0 set allowaccess ping https ssh snmp http set device-identification enable set role lan set snmp-index 28 set interface "internal5" set vlanid 3 next edit "npu0_vlink4" set vdom "root" set vrf 4 set ip 172.16.201.4 255.255.255.0 set allowaccess ping https ssh snmp http set device-identification enable set role lan set snmp-index 29 set interface "internal4" set vlanid 4 next end

[Toshi_Esumi]

npuX_vlinks are physical interfaces based on your model's npu chip(s). You can't create or remove them.

Each npu vlink has only two interfaces 0 and 1, like npu0_vlink0 and npu0_vlink1. Nothing more. Then you can use 0 on one end and 1 to the other as in the example.

If you need to have more than one pair on one npu-vlink, you have to use VLAN subinterfaces on the vlink. The most common cases are bridging between VDOMs, like connecting root vdom to all other vdoms, which could be many. In that case one VLAN interface on npu0_vlink0 is on root vdom side, another VLAN interface with the same VLAN ID on npu0_vlank1 is on the other vdom side. Then they can communicate each other over the VLAN after you put proper routing&policy in place.

[Kangming]

Hi Virg, 

 

In my lab environment, One VLINK can only meet the route leaking of 2 VRFs. If all 5 VRFs need to route leaking, that need to establish a FULL-MESH V-LINK(10 * V-LINK/10 * carry VLAN-TAG interface by VLINK) to meet the requirements, or use a normal physical interface and cooperate with an external switch.

The configuration of the second inter-connected VRF line is not correct, and it cannot communicate with different VLAN-IDs.

    edit "npu0_vlink3"     ----This is actually a VLAN interface, not a V-LINK interface

    set vdom "root"

    set vrf 3

    set ip 172.16.201.3 255.255.255.0

    set allowaccess ping https ssh snmp http

    set device-identification enable

    set role lan

    set snmp-index 28

    set interface "internal5"

    set vlanid 3

    next

    edit "npu0_vlink4"

    set vdom "root"

    set vrf 4

    set ip 172.16.201.4 255.255.255.0

    set allowaccess ping https ssh snmp http

    set device-identification enable

    set role lan

    set snmp-index 29

    set interface "internal4"

    set vlanid 4

    next

    end

For example, the following interconnection method:

internal5 --------Direct interconnection------------- internal4

internal5_tag-vlan-3 (VRF 3)-----------------------------internal4_tag-vlan-3(VRF4)

[virg9454]

Using an external switch is a last resort.    Really need this to work on box, do you have a sample config on how to configure FULL-MESH V-LINK(10 * V-LINK/10 * carry VLAN-TAG interface by VLINK?  

[Kangming]

Hi vir,

Here is my vrf lab env.

The full-mesh is too complicated, I designed multiple VRFs into a star structure：

 

My configuration：

config system global
    set vdom-mode multi-vdom
end
 
config vdom
  edit root
 
config system settings
    set allow-subnet-overlap enable
    set gui-dynamic-routing enable
    set gui-allow-unnamed-policy enable
    set gui-multiple-interface-policy enable
end
 
config system interface
    edit "vlink0_Vlan_10"
        set vdom "root"
        set vrf 10
        set ip 10.1.1.1 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink0_Vlan_10"
        set role lan
        set interface "npu0_vlink0"
        set vlanid 10
    next
    edit "vlink1_Vlan_10"
        set vdom "root" 
        set vrf 31
        set ip 10.1.1.2 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink1_Vlan_10"
        set role lan
        set interface "npu0_vlink1"
        set vlanid 10
    next
 
    edit "vlink0_Vlan_11"
        set vdom "root"
        set vrf 11
        set ip 11.1.1.1 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink0_Vlan_11"
        set role lan
        set interface "npu0_vlink0"
        set vlanid 11
    next
    edit "vlink1_Vlan_11"
        set vdom "root" 
        set vrf 31
        set ip 11.1.1.2 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink1_Vlan_11"
        set role lan
        set interface "npu0_vlink1"
        set vlanid 11
    next
 
    edit "vlink0_Vlan_12"
        set vdom "root"
        set vrf 12
        set ip 12.1.1.1 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink0_Vlan_12"
        set role lan
        set interface "npu0_vlink0"
        set vlanid 12
    next
    edit "vlink1_Vlan_12"
        set vdom "root" 
        set vrf 31
        set ip 12.1.1.2 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink1_Vlan_12"
        set role lan
        set interface "npu0_vlink1"
        set vlanid 12
    next
 
    edit "vlink0_Vlan_13"
        set vdom "root"
        set vrf 13
        set ip 13.1.1.1 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink0_Vlan_13"
        set role lan
        set interface "npu0_vlink0"
        set vlanid 13
    next
    edit "vlink1_Vlan_13"
        set vdom "root" 
        set vrf 31
        set ip 13.1.1.2 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink1_Vlan_13"
        set role lan
        set interface "npu0_vlink1"
        set vlanid 13
    next
 
    edit "vlink0_Vlan_14"
        set vdom "root"
        set vrf 14
        set ip 14.1.1.1 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink0_Vlan_14"
        set role lan
        set interface "npu0_vlink0"
        set vlanid 14
    next
    edit "vlink1_Vlan_14"
        set vdom "root" 
        set vrf 31
        set ip 14.1.1.2 255.255.255.252
        set allowaccess ping https ssh http
        set alias "vlink1_Vlan_14"
        set role lan
        set interface "npu0_vlink1"
        set vlanid 14
    next
end
 
config system zone
    edit "Core-VRF-Router"
        set intrazone allow
        set interface "vlink1_Vlan_10" "vlink1_Vlan_11" "vlink1_Vlan_12" "vlink1_Vlan_13" "vlink1_Vlan_14"
    next
end
 
config system interface
    edit "Internal_VRF10"
        set vdom "root"
        set vrf 10
        set ip 172.16.10.1 255.255.255.0
        set allowaccess ping https ssh http
        set alias "Internal_VRF10"
        set role lan
        set interface "internal"
        set vlanid 10
    next
    edit "Internal_VRF11"
        set vdom "root"
        set vrf 11
        set ip 172.16.11.1 255.255.255.0
        set allowaccess ping https ssh http
        set alias "Internal_VRF11"
        set role lan
        set interface "internal"
        set vlanid 11
    next
    edit "Internal_VRF12"
        set vdom "root"
        set vrf 12
        set ip 172.16.12.1 255.255.255.0
        set allowaccess ping https ssh http
        set alias "Internal_VRF12"
        set role lan
        set interface "internal"
        set vlanid 12
    next
    edit "Internal_VRF13"
        set vdom "root"
        set vrf 13
        set ip 172.16.13.1 255.255.255.0
        set allowaccess ping https ssh http
        set alias "Internal_VRF13"
        set role lan
        set interface "internal"
        set vlanid 13
    next
    edit "Internal_VRF14"
        set vdom "root"
        set vrf 14
        set ip 172.16.14.1 255.255.255.0
        set allowaccess ping https ssh http
        set alias "Internal_VRF14"
        set role lan
        set interface "internal"
        set vlanid 14
    next
end
 
config router static
    edit 0
        set dst 172.16.10.0 255.255.255.0
        set gateway 10.1.1.1
        set device "vlink1_Vlan_10"
    next
    edit 0
        set dst 172.16.11.0 255.255.255.0
        set gateway 11.1.1.1
        set device "vlink1_Vlan_11"
    next
    edit 0
        set dst 172.16.12.0 255.255.255.0
        set gateway 12.1.1.1
        set device "vlink1_Vlan_12"
    next
    edit 0
        set dst 172.16.13.0 255.255.255.0
        set gateway 13.1.1.1
        set device "vlink1_Vlan_13"
    next
    edit 0
        set dst 172.16.14.0 255.255.255.0
        set gateway 14.1.1.1
        set device "vlink1_Vlan_14"
    next
end
 
config firewall policy
    edit 0
        set name "any_to_core_vrf31"
        set srcintf "any"
        set dstintf "Core-VRF-Router"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
    edit 0
        set name "core_vrf31_to_any"
        set srcintf "Core-VRF-Router"
        set dstintf "any"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
        set logtraffic all
    next
end
 
FortiGate-401E (root) # get router info routing-table all
Codes: K - kernel, C - connected, S - static, R - RIP, B - BGP
       O - OSPF, IA - OSPF inter area
       N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
       E1 - OSPF external type 1, E2 - OSPF external type 2
       i - IS-IS, L1 - IS-IS level-1, L2 - IS-IS level-2, ia - IS-IS inter area
       * - candidate default
 
Routing table for VRF=0
C 10.6.30.0/24 is directly connected, mgmt
 
Routing table for VRF=10
C 10.1.1.0/30 is directly connected, vlink0_Vlan_10
C 172.16.10.0/24 is directly connected, Internal_VRF10
 
Routing table for VRF=11
C 11.1.1.0/30 is directly connected, vlink0_Vlan_11
C 172.16.11.0/24 is directly connected, Internal_VRF11
 
Routing table for VRF=12
C 12.1.1.0/30 is directly connected, vlink0_Vlan_12
C 172.16.12.0/24 is directly connected, Internal_VRF12
 
Routing table for VRF=13
C 13.1.1.0/30 is directly connected, vlink0_Vlan_13
C 172.16.13.0/24 is directly connected, Internal_VRF13
 
Routing table for VRF=14
C 14.1.1.0/30 is directly connected, vlink0_Vlan_14
C 172.16.14.0/24 is directly connected, Internal_VRF14
 
Routing table for VRF=31
C 10.1.1.0/30 is directly connected, vlink1_Vlan_10
C 11.1.1.0/30 is directly connected, vlink1_Vlan_11
C 12.1.1.0/30 is directly connected, vlink1_Vlan_12
C 13.1.1.0/30 is directly connected, vlink1_Vlan_13
C 14.1.1.0/30 is directly connected, vlink1_Vlan_14
S 172.16.10.0/24 [10/0] via 10.1.1.1, vlink1_Vlan_10
S 172.16.11.0/24 [10/0] via 11.1.1.1, vlink1_Vlan_11
S 172.16.12.0/24 [10/0] via 12.1.1.1, vlink1_Vlan_12
S 172.16.13.0/24 [10/0] via 13.1.1.1, vlink1_Vlan_13
S 172.16.14.0/24 [10/0] via 14.1.1.1, vlink1_Vlan_14
FortiGate-401E (root) #

A design such as VRF will have a problem of complex strategy configuration. Strategies and routing will be more complex, which is not conducive to maintenance and troubleshooting. More often we use VDOM should be better than VRF. VRF is more suitable for that simple routing isolation scenario.

[virg9454]

Thank you, so much Kangming.  I would really like to send you a small thank-you as you solved my issue that I have spent many hours trying to figure it out.  

VDOM has a cost and limitation implications, that is why we are not using them.  

My issue was mainly that I had not enabled VDOM so I was not getting the npu0 link available.   Seeing your configuration solved my issue. Yeah!!!

I was had to use the same VLAN ID on both sides of the npu0 link or else it would not work for me.

Here is what I did.

#global_vdom=0:vd_name=root/root

config system interface edit "wan1" set vdom "root" set mode dhcp set allowaccess ping fgfm set type physical set role wan set snmp-index 1 next edit "internal3" set vdom "root" set vrf 3 set ip 3.3.3.1 255.255.255.0 set allowaccess ping https ssh http set type physical set device-identification enable set lldp-transmission enable set role lan set snmp-index 24 next edit "internal4" set vdom "root" set vrf 4 set ip 4.4.4.1 255.255.255.0 set allowaccess ping https ssh snmp http set type physical set device-identification enable set lldp-transmission enable set role lan set snmp-index 23 next edit "internal5" set vdom "root" set vrf 5 set ip 5.5.5.1 255.255.255.0 set allowaccess ping https ssh snmp http fgfm set type physical set snmp-index 22 next edit "npu0_vlink0" set vdom "root" set type physical set snmp-index 25 next edit "npu0_vlink1" set vdom "root" set type physical set snmp-index 26 next edit "internal" set vdom "root" set ip 192.168.1.99 255.255.255.0 set allowaccess ping https ssh http fgfm fabric set type hard-switch set stp enable set role lan set snmp-index 6 next edit "vlink0_Vlan_3" set vdom "root" set vrf 3 set ip 10.1.1.1 255.255.255.252 set allowaccess ping https ssh http set status down set alias "vlink0_Vlan_3" set device-identification enable set role lan set snmp-index 27 set interface "npu0_vlink0" set vlanid 3 next edit "vlink1_Vlan_4" set vdom "root" set vrf 4 set ip 10.1.1.2 255.255.255.252 set allowaccess ping https ssh http set alias "vlink1_Vlan_4" set device-identification enable set role lan set snmp-index 28 set interface "npu0_vlink1" set vlanid 4 next edit "vlink0_Vlan_5" set vdom "root" set vrf 5 set ip 11.1.1.2 255.255.255.252 set allowaccess ping https ssh http set alias "vlink0_Vlan_4" set device-identification enable set role lan set snmp-index 29 set interface "npu0_vlink0" set vlanid 5 next edit "vlink1_Vlan_1" set vdom "root" set ip 11.1.1.2 255.255.255.252 set allowaccess ping https ssh http set status down set alias "vlink1_Vlan_1" set device-identification enable set role lan set snmp-index 30 set interface "npu0_vlink1" set vlanid 11 next edit "TestNPU" set vdom "root" set vrf 3 set ip 10.1.1.1 255.255.255.252 set allowaccess ping https ssh http set device-identification enable set role lan set snmp-index 31 set interface "npu0_vlink0" set vlanid 4 next edit "VLINK1-5" set vdom "root" set ip 11.1.1.1 255.255.255.252 set device-identification enable set role lan set snmp-index 32 set interface "npu0_vlink1" set vlanid 5 next edit "VRF5-4" set vdom "root" set vrf 5 set ip 11.1.2.2 255.255.255.252 set allowaccess ping https ssh snmp http set device-identification enable set role lan set snmp-index 33 set interface "npu0_vlink1" set vlanid 54 next edit "VRF4-5" set vdom "root" set vrf 4 set ip 11.1.2.1 255.255.255.252 set device-identification enable set role lan set snmp-index 34 set interface "npu0_vlink0" set vlanid 54 next edit "VRF3-1-VLINK" set vdom "root" set vrf 3 set ip 11.1.3.2 255.255.255.252 set device-identification enable set role lan set snmp-index 35 set interface "npu0_vlink1" set vlanid 31 next edit "VRF-1-3-VLINK" set vdom "root" set ip 11.1.3.1 255.255.255.252 set device-identification enable set role lan set snmp-index 36 set interface "npu0_vlink0" set vlanid 31 next

config system zone edit "MYVLINKS" set intrazone allow set interface "npu0_vlink0" "npu0_vlink1" "vlink0_Vlan_3" "vlink0_Vlan_5" "vlink1_Vlan_1" "vlink1_Vlan_4" "TestNPU" "VLINK1-5" "VRF4-5" "VRF5-4" "VRF-1-3-VLINK" "VRF3-1-VLINK" next end

config firewall Policy edit 10 set name "Any2VLINK" set uuid 57ce32c6-b0de-51eb-9635-31565acd8e43 set srcintf "any" set dstintf "MYVLINKS" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set logtraffic all next edit 11 set name "VLINks-2-any" set uuid 6fc16f2e-b0de-51eb-20b2-91599f7e9016 set srcintf "MYVLINKS" set dstintf "any" set srcaddr "all" set dstaddr "all" set action accept set schedule "always" set service "ALL" set logtraffic all next end

config router static edit 1 set gateway 192.168.8.1 set distance 210 set device "wan2" next edit 2 set dst 3.3.3.0 255.255.255.0 set gateway 10.1.1.1 set device "vlink1_Vlan_4" next edit 3 set dst 4.4.4.0 255.255.255.0 set gateway 10.1.1.2 set device "TestNPU" next edit 4 set gateway 11.1.1.1 set device "vlink0_Vlan_5" next edit 5 set dst 5.5.5.0 255.255.255.0 set gateway 11.1.1.2 set device "VLINK1-5" next edit 6 set dst 5.5.5.0 255.255.255.0 set gateway 11.1.2.2 set device "VRF4-5" next edit 7 set dst 4.4.4.0 255.255.255.0 set gateway 11.1.2.1 set device "VRF5-4" next edit 8 set gateway 11.1.3.1 set device "VRF3-1-VLINK" next edit 9 set dst 3.3.3.0 255.255.255.0 set gateway 11.1.3.2 set device "VRF-1-3-VLINK" next end

[Kangming]

You're welcome, Virg, Glad to hear this message.