Technical Tip: VXLAN with SD-WAN

alif · ‎12-23-2019

Description
This article describes how to extend VLANs (VXLAN) over multiple WAN connections (SD-WAN).

Solution
Virtual Extensible LAN (VXLAN) is a network virtualization technology used in large cloud computing deployments.
It encapsulates OSI layer 2 Ethernet frames within layer 3 IP packets using standard destination port 4789.

SD-WAN allows to load balance traffic between multiple WAN connections and thereby providing redundancy when one of the WAN connection is unavailable.

In this scenario, FortiGate has multiple WAN connections which are configured under SD-WAN interface.
Multiple VLANs (same at both locations) are configured.
VXLAN connects locations at Layer2 over Layer3.

- WAN Configuration.

As shown on the diagram, both FortiGates have multiple WAN connections namely wan1 and wan2.

FortiGate 1

FortiGate 2

# config system interface
    edit "wan1"
        set vdom "root"
        set ip 11.11.11.11 255.255.255.0
        set allowaccess ping fgfm
        set type physical
        set lldp-reception enable
        set role wan
        set snmp-index 1
    next
    edit "wan2"
        set vdom "root"
        set ip 33.33.33.33 255.255.255.0
        set allowaccess ping fgfm
        set type physical
        set lldp-reception enable
        set role wan
        set snmp-index 2
    next
end

# config system interface
    edit "wan1"
        set vdom "root"
        set ip 22.22.22.22 255.255.255.0
        set allowaccess ping fgfm
        set type physical
        set lldp-reception enable
        set role wan
        set snmp-index 1
    next
    edit "wan2"
        set vdom "root"
        set ip 44.44.44.44 255.255.255.0
        set allowaccess ping fgfm
        set type physical
        set lldp-reception enable
        set role wan
        set snmp-index 2
    next
end

- Loopback Interface.

A loopback interface is configured at each location.

# config system interface
    edit "Lo1"
        set vdom "root"
        set ip 1.1.1.1 255.255.255.255
        set allowaccess ping https http
        set type loopback
        set role lan
        set snmp-index 11
    next
end

# config system interface
    edit "Lo1"
        set vdom "root"
        set ip 2.2.2.2 255.255.255.255
        set allowaccess ping https http
        set type loopback
        set role lan
        set snmp-index 11
    next
end

- VXLAN Interface.

A VXLAN interface is configured as a bound to the loopback interface.
The remote IP configured under system.vxlan is the peer side loopback interface IP address.

# config system vxlan
    edit "vxlan1"
        set interface "Lo1"
        set vni 10
        set remote-ip "2.2.2.2"
    next
end

# config system vxlan
    edit "vxlan1"
        set interface "Lo1"
        set vni 10
        set remote-ip "1.1.1.1"
    next
end

- SD-WAN Interface.

A SD-WAN interface is used under wan1 and wan2 defined as members.

# config system virtual-wan-link
    set status enable
    config members
        edit 1
            set interface "wan1"
            set gateway 11.11.11.1
        next
        edit 2
            set interface "wan2"
            set gateway 33.33.33.3
        next
    end
end

# config system virtual-wan-link
    set status enable
    config members
        edit 1
            set interface "wan1"
            set gateway 22.22.22.2
        next
        edit 2
            set interface "wan2"
            set gateway 44.44.44.4
        next
    end
end

- Static Route.

A single static route is configured to send all the traffic outside via SD-WAN interface.

# config router static
    edit 1
        set distance 1
        set virtual-wan-link enable
    next
end

- Switch Interface.

A software switch is configured with port1 and the VXLAN interface 'vxlan1'.
Port1 is connected to an internal switch where multiple VLAN interfaces are configured.
In the configuration below, the 'set intra-switch-policy implicit' (default) command implicitly allows traffic between switch members port1 and VXLAN1.

# config system switch-interface
    edit "local1"
        set vdom "root"
        set member "port1" "vxlan1"
    next
end

Note that if 'set intra-switch-policy explicit' is used, it will require additional firewall policies to allow traffic between port1 and VXLAN.

- Firewall Policy.

Firewall policy is needed to allow traffic between the SD-WAN and the loopback interfaces.

# config firewall policy
    edit 1
        set srcintf "Lo1"
        set dstintf "virtual-wan-link"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
    edit 2
        set srcintf "virtual-wan-link"
        set dstintf "Lo1"
        set srcaddr "all"
        set dstaddr "all"
        set action accept
        set schedule "always"
        set service "ALL"
    next
end

- Testing.

When Host1 (VLAN10: 192.168.10.1/24) which is connected on switch1 tries to reach Host2 (VLAN10: 192.168.10.2/24) connected to switch2, the following behavior will be observed.

C:\Users\Host1>ping 192.168.10.2

Pinging 192.168.10.2 with 32 bytes of data:
Reply from 192.168.10.2: bytes=32 time=37ms TTL=50
Reply from 192.168.10.2: bytes=32 time=34ms TTL=50
Reply from 192.168.10.2: bytes=32 time=37ms TTL=50
Reply from 192.168.10.2: bytes=32 time=43ms TTL=50

Ping statistics for 192.168.10.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 34ms, Maximum = 43ms, Average = 37ms

The benefit of above setup is that if wan1 goes unreachable, traffic between the hosts at both locations will be flowing over wan2 connection.

The above scenario can also work without SD-WAN interface.
Instead of a SD-WAN interface, static routes for each WAN interface should be configured.